优化一

故障现象

radius认证存在大量的重复请求，radius server在timeout之前无法及时响应。

分析过程

1、使用java提供的线程dump工具dump出当前的线程状态，进行分析

Thread: default_authen_thread7(16408500110), State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(SocketInputStream.java:-2)
    at java.net.SocketInputStream.read(SocketInputStream.java:152)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at java.io.DataInputStream.readFully(DataInputStream.java:195)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:846)
    at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:727)
    at net.sourceforge.jtds.jdbc.ResponseStream.getPacket(ResponseStream.java:466)
    at net.sourceforge.jtds.jdbc.ResponseStream.read(ResponseStream.java:103)
    at net.sourceforge.jtds.jdbc.ResponseStream.peek(ResponseStream.java:88)
    at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3932)
    at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1046)
    at net.sourceforge.jtds.jdbc.TdsCore.submitSQL(TdsCore.java:899)
    at net.sourceforge.jtds.jdbc.ConnectionJDBC2.commit(ConnectionJDBC2.java:2094)
    at com.alibaba.druid.filter.FilterChainImpl.connection_commit(FilterChainImpl.java:180)
    at com.alibaba.druid.filter.FilterAdapter.connection_commit(FilterAdapter.java:780)
    at com.alibaba.druid.filter.FilterChainImpl.connection_commit(FilterChainImpl.java:176)
    at com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl.commit(ConnectionProxyImpl.java:123)
    at com.alibaba.druid.pool.DruidPooledConnection.commit(DruidPooledConnection.java:731)
    at org.hibernate.transaction.JDBCTransaction.commitAndResetAutoCommit(JDBCTransaction.java:170)
    at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:146)
    at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:655)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
    at com.ruijie.spl.common.cache2.TransactionManagerWrapper.commit(TransactionManagerWrapper.java:42)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at com.sun.proxy.$Proxy412.addMacBindInfoTCache(null:-1)
    at com.ruijie.spl.usermanage.service.UserMabCheckServiceImpl.autoStudyTerminalInfo(UserMabCheckServiceImpl.java:249)
    at com.ruijie.spl.radius.author.face.AuthorInterface.verifyForPackageMode(AuthorInterface.java:2860)
    at com.ruijie.spl.radius.author.face.AuthorInterface.verify(AuthorInterface.java:257)
    at com.ruijie.spl.radius.authen.support.eap.AuthenRuleImpl_EAP.doEapMD5Authen(AuthenRuleImpl_EAP.java:374)
    at com.ruijie.spl.radius.authen.support.eap.AuthenRuleImpl_EAP.doAuthen(AuthenRuleImpl_EAP.java:308)
    at com.ruijie.spl.radius.authen.support.RadiusAuthenRule.doAuthen(RadiusAuthenRule.java:56)
    at com.ruijie.spl.radius.authen.process.RadiusProcessor.process(RadiusProcessor.java:490)
    at com.ruijie.spl.common.net.radius.PacketProcessThread.runOnce(PacketProcessThread.java:68)
    at com.ruijie.spl.common.power.AbstractDaemonThread.run(AbstractDaemonThread.java:157)

分析发现，当前的性能瓶颈在数据库，认证过程中发生了数据库的IO操作。
分析代码发现，实质上当前卡住的位置使用的是异步数据库插入、更新策略，不存在数据库IO。

public void update(MacBindInfo info, boolean async) {
        UpdateMacBindInfoTask updateTask = new UpdateMacBindInfoTask(this, info, getTransactionTemplate().getTransactionManager());
        AsyncExecutorManager asyncManager = (AsyncExecutorManager)SpringBeanFinder.findSingleton("asyncExecutorManager");
        if (async && asyncManager.isEnabled()) {
            asyncManager.getExecutor().submit(updateTask);//程序中实质上使用的异步队列
        } else {
            updateTask.run();
        }
        if (Debuger.isDebugEnabled()) {
            Debuger.debug("用户: "+info.getUserId()+" 已经被刷新到数据库");
        }
    }

仔细分析发现，spring的申明式事务管理，对接口进行了动态代理，自动加上了开启事务、提交事务的操作，涉及到数据库IO操作。

<bean id="transactionProxy" abstract="true"
        class="org.springframework.transaction.interceptor.TransactionProxyFactoryBean">
        <property name="transactionManager" ref="TransactionManager" />
        <property name="transactionAttributes">
            <props>
                <prop key="move*">PROPAGATION_REQUIRED,-Exception</prop>
                <prop key="insert*">
                    PROPAGATION_REQUIRED,-Exception
                </prop>
                <prop key="add*">PROPAGATION_REQUIRED,-Exception</prop>
                <prop key="delete*">
                    PROPAGATION_REQUIRED,-Exception
                </prop>
                <prop key="update*">
                    PROPAGATION_REQUIRED,-Exception
                </prop>
                <prop key="save*">PROPAGATION_REQUIRED,-Exception</prop>
                <prop key="find*">PROPAGATION_SUPPORTS,-readOnly</prop>
                <prop key="get*">PROPAGATION_SUPPORTS,-readOnly</prop>
                <prop key="count*">PROPAGATION_SUPPORTS,-readOnly</prop>
                <prop key="check*">PROPAGATION_SUPPORTS,-readOnly</prop>
            </props>
        </property>
    </bean>

业务的解法：分析业务发现，该业务设计的初衷是必须使用异步提交策略，不应该开启事务，结合spring的申明式事务配置机制，对接口重新定义（换个名字）。

2、进行如上优化后，故障并没有明显的缓解，继续进行线程dump分析。

Thread: default_authen_thread31(17414000170), State: WAITING
    at sun.misc.Unsafe.park(Unsafe.java:-2)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
    at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
    at com.ruijie.spl.proxy.pool.StrategySamMapPool.findAll(StrategySamMapPool.java:104)
    at com.ruijie.spl.proxy.service.StrategySamServiceImpl.getStrategyAllByOperatorNmae(StrategySamServiceImpl.java:61)
    at sun.reflect.GeneratedMethodAccessor1891.invoke(null:-1)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at com.sun.proxy.$Proxy467.getStrategyAllByOperatorNmae(null:-1)operater.strategy.autoregist
    at com.ruijie.spl.proxy.util.StrategyCheckUtil.checkLimitemode(StrategyCheckUtil.java:37)
    at com.ruijie.spl.proxy.mapping.AcceptIntercepter.processAccept(AcceptIntercepter.java:225)
    at com.ruijie.spl.proxy.util.ProxyUtil.acceptIntercept(ProxyUtil.java:1868)
    at com.ruijie.spl.proxy.util.ProxyUtil.processLocalResponse(ProxyUtil.java:1670)
    at com.ruijie.spl.radius.authen.process.RadiusProcessor.authProxyResponseProcess(RadiusProcessor.java:1529)
    at com.ruijie.spl.radius.authen.process.RadiusProcessor.process(RadiusProcessor.java:404)
    at com.ruijie.spl.common.net.radius.PacketProcessThread.runOnce(PacketProcessThread.java:68)
    at com.ruijie.spl.common.power.AbstractDaemonThread.run(AbstractDaemonThread.java:157)

分析发现，认证线程都卡在这个锁的地方，分析代码，代码有个查找全部的方法，加了一个读写锁的读锁。

public List<StrategySam> findAll() {
        lock.readLock().lock();
        try {
            LinkedList<StrategySam> result = new LinkedList<StrategySam>();
            if(null!=strategySamMap.values()&& strategySamMap.values().size()>0){
                result.addAll(strategySamMap.values());
            }
            return result;
        } finally {
            lock.readLock().unlock();
        }
        
    }

查找整个线程dump，写锁并没有锁住，对于读写锁，在没有写锁的情况，读锁多线程可以共享，不应该存在性能瓶颈问题。
findall接口，查找全部并遍历，因为问题在这个地方，看出数据库中的数据，表中存在大量数据。

解法：算法优化。

故障二

故障现象

rsa加解密算法性能瓶颈

Thread: PeapEapAuthProcessorThread-19, State: BLOCKED
    at sun.security.rsa.RSACore$BlindingParameters.getBlindingRandomPair(RSACore.java:404)
    at sun.security.rsa.RSACore.getBlindingRandomPair(RSACore.java:445)
    at sun.security.rsa.RSACore.crtCrypt(RSACore.java:178)
    at sun.security.rsa.RSACore.rsa(RSACore.java:122)
    at sun.security.rsa.RSASignature.engineSign(RSASignature.java:177)
    at java.security.Signature$Delegate.engineSign(Signature.java:1205)
    at java.security.Signature.sign(Signature.java:578)
    at sun.security.ssl.HandshakeMessage$ECDH_ServerKeyExchange.<init>(HandshakeMessage.java:1012)
    at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:779)
    at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:167)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:913)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:853)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:851)
    at java.security.AccessController.doPrivileged(AccessController.java:-2)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1285)
    at com.ruijie.spl.radius.authen.support.tls.TlsEngine.runSslTask(TlsEngine.java:278)
    at com.ruijie.spl.radius.authen.support.tls.TlsEngine.doSSLHandshake(TlsEngine.java:217)
    at com.ruijie.spl.radius.authen.support.tls.TlsEngine.process(TlsEngine.java:139)
    at com.ruijie.spl.radius.authen.support.peap.PeapAuthEngine.doPeap(PeapAuthEngine.java:199)
    at com.ruijie.spl.radius.authen.support.peap.PeapAuthEngine.process(PeapAuthEngine.java:81)
    at com.ruijie.spl.radius.authen.support.peap.PeapAuthEngine.process(PeapAuthEngine.java:70)
    at com.ruijie.spl.radius.authen.support.eap.EapAuthEngine.doInnerAuth(EapAuthEngine.java:176)
    at com.ruijie.spl.radius.authen.support.eap.EapAuthEngine.doAuth(EapAuthEngine.java:136)
    at com.ruijie.spl.radius.authen.support.eap.EapAuthEngine.doAuth(EapAuthEngine.java:59)
    at com.ruijie.spl.radius.authen.support.eap.EapAuthEngine.doAuthen(EapAuthEngine.java:64)
    at com.ruijie.spl.radius.authen.support.eap.AuthenRuleImpl_EAP_PEAP.doAuthen(AuthenRuleImpl_EAP_PEAP.java:136)
    at com.ruijie.spl.radius.authen.support.RadiusAuthenRule.doAuthen(RadiusAuthenRule.java:58)
    at com.ruijie.spl.radius.authen.support.peap.PeapEapAuthProcessorThread.runOnce(PeapEapAuthProcessorThread.java:166)
    at com.ruijie.spl.common.power.AbstractDaemonThread.run(AbstractDaemonThread.java:156)

分析发现，RSA算法加解密需要进行大量计算，JDK底层针对这个问题进行的优化策略是引入缓存。引入缓存后，为解决多线程并发计算的现场安全问题，在计算时进行加锁处理。导致了本次的性能瓶颈

优化方案

由于在业务场景下（无线1x认证，TLS通道加密），必须是RSA算法。同时在网上搜索许久也没有搜到源码。想是JDK真的这部分没有开放源代码。最终想到的是使用反射进行缓存替换，使用线程本地变量，将缓存进行扩充，解决多线程锁的性能消耗。

      final Wireless1XWeakHashMap Wireless1XWeakHashMap = new Wireless1XWeakHashMap();
        Field nameField1 = RSACore.class.getDeclaredField("blindingCache");
        
        Field modifiersField1 = Field.class.getDeclaredField("modifiers"); 
        modifiersField1.setAccessible(true);
        modifiersField1.setInt(nameField1, nameField1.getModifiers() & ~Modifier.FINAL); //解决final无法重新赋值的问题
        
        nameField1.setAccessible(true); 
        nameField1.set(RSACore.class, Wireless1XWeakHashMap);
        System.out.println(nameField1.get(RSACore.class)); 

public class Wireless1XWeakHashMap extends WeakHashMap<BigInteger, Object>{
    /***存储 RSACore。BlindingParameters 类型***/
    ThreadLocal<WeakHashMap<BigInteger, Object>> longLocal = new ThreadLocal<WeakHashMap<BigInteger, Object>>();
    
    public Object get(BigInteger key) {
        WeakHashMap<BigInteger, Object> weakHashMap = longLocal.get();
        if(weakHashMap == null) {
            return null;
        }
        
        return weakHashMap.get(key);
    }
    
    
    public Object put(BigInteger key, Object value) {
        WeakHashMap<BigInteger, Object> weakHashMap = longLocal.get();
        if(weakHashMap == null) {
            longLocal.set(new WeakHashMap<BigInteger, Object>());
        }
        weakHashMap = longLocal.get();
        return weakHashMap.put(key, value);
    }
}

参考：http://blog.51cto.com/nxlhero/1832127

jtds + sqlserver2008r2的一次数据库连接被持续关闭的故障定位

故障现象

系统在运行时，整个系统的业务不定时不正常，包括web页面的访问，在不正常时，后台对应的错误日志为数据库连接被关闭

2018-10-09 20:49:50,029 ERROR [default_authen_thread10] com.ruijie.spl.radius.authen.process.RadiusProcessor - 
org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is org.hibernate.TransactionException: JDBC rollback failed
    at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:677)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:823)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:800)
    at com.ruijie.spl.common.cache2.TransactionManagerWrapper.rollback(TransactionManagerWrapper.java:84)
    at org.springframework.transaction.support.TransactionTemplate.rollbackOnException(TransactionTemplate.java:154)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:132)
    at com.ruijie.spl.onlineuser.service.OnlineUserServiceImpl.directDeleteAndAuthLater(OnlineUserServiceImpl.java:2664)
    at sun.reflect.GeneratedMethodAccessor2565.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
    at com.sun.proxy.$Proxy196.directDeleteAndAuthLater(Unknown Source)
    at com.ruijie.spl.radius.authen.support.DirectDeleteFalseOnlineUserAction.execution(DirectDeleteFalseOnlineUserAction.java:13)
    at com.ruijie.spl.radius.authen.process.RadiusProcessor.process(RadiusProcessor.java:689)
    at com.ruijie.spl.common.net.radius.PacketProcessThread.runOnce(PacketProcessThread.java:62)
    at com.ruijie.spl.common.power.AbstractDaemonThread.run(AbstractDaemonThread.java:156)
Caused by: org.hibernate.TransactionException: JDBC rollback failed
    at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:204)
    at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:674)
    ... 16 more
Caused by: java.sql.SQLException: Invalid state, the Connection object is closed.
    at net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1699)
    at net.sourceforge.jtds.jdbc.ConnectionJDBC2.rollback(ConnectionJDBC2.java:2100)
    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.jdbcRollback(BaseWrapperManagedConnection.java:697)
    at org.jboss.resource.adapter.jdbc.WrappedConnection.rollback(WrappedConnection.java:500)
    at com.ruijie.spl.common.datasource.SplConnection.rollback(SplConnection.java:196)
    at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:217)
    at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:196)
    ... 17 more

故障分析

1、查看源码发现，jtds的数据库连接的状态通过boolean 成员变量closed进行控制，在网上查找资料，说1.2.5版本存在一些问题，于是借着这个机会将jtds的版本进行了一次升级。

2、下载jtds的1.3.1的源码进行分析，发现源码变化很大，原本的net.sourceforge.jtds.jdbc.ConnectionJDBC2已经变更为了net.sourceforge.jtds.jdbc.JtdsConnection。分析下源码，将关闭数据库连接的地方都加上调试日志
void setClosed()
synchronized public void close() throws SQLException

3、升级jtds到1.3.1，故障现象依旧，问题依然没有得到解决，但是本地升级机制的将日志加上了，开启log4j的日志，日志报错信息如下

2018-10-15 20:02:02,131 ERROR [default_authen_thread7] net.sourceforge.jtds.jdbc.JtdsConnection - ------------数据库被setClosed关闭-----------
java.lang.Exception
    at net.sourceforge.jtds.jdbc.JtdsConnection.setClosed(JtdsConnection.java:1855)
    at net.sourceforge.jtds.jdbc.TdsCore.tdsDoneToken(TdsCore.java:3698)
    at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2472)
    at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:677)
    at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:613)
    at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:572)
    at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:727)
    at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeUpdate(CachedPreparedStatement.java:96)
    at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:365)
    at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:798)
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591)
    at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:792)
    at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:850)
    at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:858)
    at com.ruijie.spl.onlineuser.dao.OnlineUserDAOImpl.delete(OnlineUserDAOImpl.java:813)
    at com.ruijie.spl.onlineuser.service.OnlineUserServiceImpl.deleteOnlineUser(OnlineUserServiceImpl.java:396)
    at com.ruijie.spl.onlineuser.service.OnlineUserServiceImpl.dealUserLogout(OnlineUserServiceImpl.java:2481)
    at com.ruijie.spl.onlineuser.service.OnlineUserServiceImpl.billAndLogout(OnlineUserServiceImpl.java:2230)
    at sun.reflect.GeneratedMethodAccessor915.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
    at com.sun.proxy.$Proxy196.billAndLogout(Unknown Source)
    at com.ruijie.spl.onlineuser.interf.OnlineUserForRadiusService.billAndLogout(OnlineUserForRadiusService.java:79)
    at com.ruijie.spl.radius.author.face.InterfFromOnlineuserService$1.doInTransaction(InterfFromOnlineuserService.java:113)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
    at com.ruijie.spl.radius.author.face.InterfFromOnlineuserService.billAndLogout(InterfFromOnlineuserService.java:111)
    at com.ruijie.spl.onlineuser.service.OnlineUserServiceImpl$4.doInTransaction(OnlineUserServiceImpl.java:2669)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
    at com.ruijie.spl.onlineuser.service.OnlineUserServiceImpl.directDeleteAndAuthLater(OnlineUserServiceImpl.java:2664)
    at sun.reflect.GeneratedMethodAccessor984.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
    at com.sun.proxy.$Proxy196.directDeleteAndAuthLater(Unknown Source)
    at com.ruijie.spl.radius.authen.support.DirectDeleteFalseOnlineUserAction.execution(DirectDeleteFalseOnlineUserAction.java:13)
    at com.ruijie.spl.radius.authen.process.RadiusProcessor.process(RadiusProcessor.java:689)
    at com.ruijie.spl.common.net.radius.PacketProcessThread.runOnce(PacketProcessThread.java:62)
    at com.ruijie.spl.common.power.AbstractDaemonThread.run(AbstractDaemonThread.java:156)

根据日志信息，结合系统业务，当前执行的sql为

"DELETE FROM ONLINE_USER WHERE ONLINE_USER_UUID=?", new Object[]{onlineUser.getOnlineUserUuid()}

分析sqlserver的日志,服务器上会相对应的记录如下错误信息

日期        2018/10/16 11:35:46
日志        SQL Server (存档编号1 - 2018/10/16 13:14:00)

源        spid57

消息
Database ID 5, Page (1:1729) is marked RestorePending, which may indicate disk corruption. To recover from this state, perform a restore.

说明数据库完整性存在问题。使用如下脚本进行完整性修复

exec sp_dboption DB_, 'single_user', true
DBCC CHECKDB (DB_NAME, repair_allow_data_loss) with NO_INFOMSGS
exec sp_dboption DB_NAME, 'single_user', false

修复完后

修复: 已为数据库 'DB_NAME' 中的对象 'dbo.ONLINE_USER, idx_online_user_nas' 成功地重新生成了 Nonclustered 索引。
修复: 页 (1:1729) 已从对象 ID 410484541，索引 ID 3，分区 ID 72057596299902976，分配单元 ID 72057596354363392 (类型为 In-row data)释放。
消息 8945，级别 16，状态 1，第 2 行
表错误: 将重新生成对象 ID 410484541，索引 ID 3。
        该错误已修复。
消息 8928，级别 16，状态 1，第 2 行
对象 ID 410484541，索引 ID 3，分区 ID 72057596299902976，分配单元 ID 72057596354363392 (类型为 In-row data): 无法处理页 (1:1729)。有关详细信息，请参阅其他错误消息。
        该错误已修复。
消息 8939，级别 16，状态 98，第 2 行
表错误: 对象 ID 410484541，索引 ID 3，分区 ID 72057596299902976，分配单元 ID 72057596354363392 (类型为 In-row data)，页 (1:1729)。测试(IS_OFF (BUF_IOERR, pBUF->bstat))失败。值为 12584969 和 -6。
        该错误已修复。
消息 8976，级别 16，状态 1，第 2 行
表错误: 对象 ID 410484541，索引 ID 3，分区 ID 72057596299902976，分配单元 ID 72057596354363392 (类型为 In-row data)。在扫描过程中未发现页 (1:1729)，但该页的父级 (1:13283) 和上一页 (1:53014) 都引用了它。请检查以前的错误消息。
        该错误已修复。
消息 8978，级别 16，状态 1，第 2 行
表错误: 对象 ID 410484541，索引 ID 3，分区 ID 72057596299902976，分配单元 ID 72057596354363392 (类型为 In-row data)。页 (1:53008) 缺少上一页 (1:1729) 对它的引用。可能是因为链链接有问题。
        该错误已修复。

至此，问题完美解决。