Dr.Elephant mysql connection error
This is the first time I try to use english to write my blog, so don't jeer at the mistake of my grammar and spelling. Because of multi threaded drelephant will cause JobHistoryServer’s Loads very high, so I stopped it for a strench of time. Until last week, a period pull from JHS patch merge request from github was released. I re-compiled dr. elephant and deploy the new dr. elephant onthe cluster. It seems stable, but on this Monday morning, my leader told me that there were no more counters and any information about cluster jobs in dr. elephant. So I logged in to the server, and check log, then I found this message below. [error]c.j.b.ConnectionHandle-Databaseaccessproblem.Killingoffthisconnectionandallremainingconnectionsintheconnectionpool.SQLState=HY000 And then, I found two things will cause this issue, one of them is selinux config is set to enforcing, change this config value to disabled and reboot the server, it seem good, but still got same error, only not too many. Then I review the code of dr. elephant, I find out that in mysql create table in mysql initializing script of play framework has an issue. the index is too long, I change the index to 250 previously, but if an url is longer then 250, it gone wrong. createindexyarn_app_result_i4onyarn_app_result(flow_exec_id(250)); createindexyarn_app_result_i5onyarn_app_result(job_def_id(250)); createindexyarn_app_result_i6onyarn_app_result(flow_def_id(250)); So I delete this index limitation, and rewrite the sql like this, add a innodb_large_prefix and row_format=dynamic to the table creation script, and finally, no more error log appears… SETGLOBALinnodb_file_format=Barracuda; SETGLOBALinnodb_large_prefix=ON; CREATETABLEyarn_app_result( idVARCHAR(50)NOTNULLCOMMENT'Theapplicationid,e.g.,application_1236543456321_1234567', nameVARCHAR(100)NOTNULLCOMMENT'Theapplicationname', usernameVARCHAR(50)NOTNULLCOMMENT'Theuserwhostartedtheapplication', queue_nameVARCHAR(50)DEFAULTNULLCOMMENT'Thequeuetheapplicationwassubmittedto', start_timeBIGINTUNSIGNEDNOTNULLCOMMENT'Thetimeinwhichapplicationstarted', finish_timeBIGINTUNSIGNEDNOTNULLCOMMENT'Thetimeinwhichapplicationfinished', tracking_urlVARCHAR(255)NOTNULLCOMMENT'ThewebURLthatcanbeusedtotracktheapplication', job_typeVARCHAR(20)NOTNULLCOMMENT'TheJobTypee.g,Pig,Hive,Spark,HadoopJava', severityTINYINT(2)UNSIGNEDNOTNULLCOMMENT'Aggregateseverityofalltheheuristics.Rangesfrom0(LOW)to4(CRITICAL)', scoreMEDIUMINT(9)UNSIGNEDDEFAULT0COMMENT'Theapplicationscorewhichisthesumofheuristicscores', workflow_depthTINYINT(2)UNSIGNEDDEFAULT0COMMENT'Theapplicationdepthinthescheduledflow.Depthstartsfrom0', schedulerVARCHAR(20)DEFAULTNULLCOMMENT'Theschedulerwhichtriggeredtheapplication', job_nameVARCHAR(255)NOTNULLDEFAULT''COMMENT'Thenameofthejobintheflowtowhichthisappbelongs', job_exec_idVARCHAR(800)NOTNULLDEFAULT''COMMENT'Auniquereferencetoaspecificexecutionofthejob/action(jobintheworkflow).Thisshouldfilterallapplications(mapreduce/spark)triggeredbythejobfora particularexecution.', flow_exec_idVARCHAR(255)NOTNULLDEFAULT''COMMENT'Auniquereferencetoaspecificflowexecution.Thisshouldfilterallapplicationsfiredbyaparticularflowexecution.Notethatiftheschedulersupportssub- workflows,thenthisIDshouldbethesuperparentflowexecutionidthattriggeredthetheapplicationsandsub-workflows.', job_def_idVARCHAR(800)NOTNULLDEFAULT''COMMENT'Auniquereferencetothejobintheentireflowindependentoftheexecution.Thisshouldfilteralltheapplications(mapreduce/spark)triggeredbythejobforal lthehistoricexecutionsofthatjob.', flow_def_idVARCHAR(800)NOTNULLDEFAULT''COMMENT'Auniquereferencetotheentireflowindependentofanyexecution.Thisshouldfilterallthehistoricmrjobsbelongingtotheflow.Notethatifyourscheduler supportssub-workflows,thenthisIDshouldreferencethesuperparentflowthattriggeredtheallthejobsandsub-workflows.', job_exec_urlVARCHAR(800)NOTNULLDEFAULT''COMMENT'Aurltothejobexecutiononthescheduler', flow_exec_urlVARCHAR(800)NOTNULLDEFAULT''COMMENT'Aurltotheflowexecutiononthescheduler', job_def_urlVARCHAR(800)NOTNULLDEFAULT''COMMENT'Aurltothejobdefinitiononthescheduler', flow_def_urlVARCHAR(800)NOTNULLDEFAULT''COMMENT'Aurltotheflowdefinitiononthescheduler', PRIMARYKEY(id) )ENGINE=InnoDBDEFAULTCHARSET=utf8ROW_FORMAT=DYNAMIC; createindexyarn_app_result_i1onyarn_app_result(finish_time); createindexyarn_app_result_i2onyarn_app_result(username,finish_time); createindexyarn_app_result_i3onyarn_app_result(job_type,username,finish_time); createindexyarn_app_result_i4onyarn_app_result(flow_exec_id); createindexyarn_app_result_i5onyarn_app_result(job_def_id); createindexyarn_app_result_i6onyarn_app_result(flow_def_id); createindexyarn_app_result_i7onyarn_app_result(start_time); CREATETABLEyarn_app_heuristic_result( idINT(11)NOTNULLAUTO_INCREMENTCOMMENT'Theapplicationheuristicresultid', yarn_app_result_idVARCHAR(50)NOTNULLCOMMENT'Theapplicationid', heuristic_classVARCHAR(255)NOTNULLCOMMENT'NameoftheJVMclassthatimplementsthisheuristic', heuristic_nameVARCHAR(128)NOTNULLCOMMENT'Theheuristicname', severityTINYINT(2)UNSIGNEDNOTNULLCOMMENT'Theheuristicseverityrangingfrom0(LOW)to4(CRITICAL)', scoreMEDIUMINT(9)UNSIGNEDDEFAULT0COMMENT'Theheuristicscorefortheapplication.score=severity*number_of_tasks(map/reduce)whereseveritynotin[0,1],otherwisescore=0', PRIMARYKEY(id), CONSTRAINTyarn_app_heuristic_result_f1FOREIGNKEY(yarn_app_result_id)REFERENCESyarn_app_result(id) ); createindexyarn_app_heuristic_result_i1onyarn_app_heuristic_result(yarn_app_result_id); createindexyarn_app_heuristic_result_i2onyarn_app_heuristic_result(heuristic_name,severity); CREATETABLEyarn_app_heuristic_result_details( yarn_app_heuristic_result_idINT(11)NOTNULLCOMMENT'Theapplicationheuristicresultid', nameVARCHAR(128)NOTNULLDEFAULT''COMMENT'Theanalysisdetailentryname/key', valueVARCHAR(255)NOTNULLDEFAULT''COMMENT'Theanalysisdetailvaluecorrespondingtothename', detailsTEXTCOMMENT'Moreinformationonanalysisdetails.e.g,stacktrace', PRIMARYKEY(yarn_app_heuristic_result_id,name), CONSTRAINTyarn_app_heuristic_result_details_f1FOREIGNKEY(yarn_app_heuristic_result_id)REFERENCESyarn_app_heuristic_result(id) ); createindexyarn_app_heuristic_result_details_i1onyarn_app_heuristic_result_details(name); At last I send a pull request to linkedin on github.com…
