{"title": "A Neural Expert System with Automated Extraction of Fuzzy If-Then Rules and Its Application to Medical Diagnosis", "book": "Advances in Neural Information Processing Systems", "page_first": 578, "page_last": 584, "abstract": null, "full_text": "A Neural Expert System with Automated Extraction \n\nof Fuzzy If-Then Rules and Its Application to \n\nMedical Diagnosis \n\nYoichi Hayashi* \n\nDepartment of Computer and Information Sciences \n\nIbaraki University \n\nHitachi-shi,Ibaraki 316, Japan \n\nABSTRACT \n\nThis paper proposes ajuzzy neural expert system (FNES) with the \nfollowing two functions: (1) Generalization of the information derived \nfrom the training data and embodiment of knowledge in the form of the \nfuzzy neural network; (2) Extraction of fuzzy If-Then rules with \nlinguistic relative importance of each proposition in an antecedent \n(I f -part) from a trained neural network. This paper also gives a \nmethod to extract automatically fuzzy If-Then rules from the trained \nneural network. To prove the effectiveness and validity of the proposed \nfuzzy neural expert system. a fuzzy neural expert system for medical \ndiagnosis has been developed. \n\n1 INTRODUCTION \n\nExpert systems that have neural networks for their knowledge bases are sometimes called \nneural expert system (Gallant & Hayashi. 1990; Hayashi et at. 1990; Yoshida et al .\u2022 \n1990) or connectionist expert system (Gallant. 1988; Yoshida et a1.. 1989). This paper \nextends work reported in (Hayashi & Nakai. 1990; Hayashi et a!.. 1990) and shows a new \nmethod to give confidence measurements for all inferences and explanations to neural \nexpert systems. In contrast with conventional expert systems. we propose ajuzzy neural \nexpert system (FNES) with automated extraction of fuzzy If-Then rules. This paper also \ngives a method to extract automatically fuzzy If-Then rules with linguistic relative \nimportance of each proposition in an antecedent (If-part) from a trained neural network. \nTo prove the effectiveness and validity of the proposed neural expen system. a fuzzy \nneural expert system for diagnosing hepatobiliary disorders has been developed by using a \nreal medical database. This paper compares the diagnostic capability provided by the \nneural network approach and that provided by the statistical approach. Furthermore. we \nevaluate the performance of extracted fuzzy If-Then rules from a neural network \nknowledge base. \n\n* A part of this work was performed when the author was with the University of Alabama at \nBirmingham, Department of Computer and Information Sciences as a Visiting Associate \nProfessor. \n\n578 \n\n\fA Neural Expert System with Automated Extraction of fuzzy If-Then Rules \n\n579 \n\n2 FUZZY NEURAL EXPERT SYSTEM WITH AUTOMATED \nEXTRACTION OF FUZZY IF-THEN RULES \n\n2.1 Distributed Neural Network \nFigure 1 illustrates a schematic diagram of a fuzzy neural expert system with automated \nextraction of fuzzy If-Then rules. For backpropagation. the configuration consisting of p \ninput cells. q intermediate cells (\"hidden units\") and r output cells has been the most \nwidely used. Connections run from every input cell to every intermediate cell. and from \nevery intermediate cell to every output cell. In this paper. we employ a valiant of \nconventional perceptron network. which is called distributed (neural) network (Gallant. \n1990). In the network. there are the same cells and connections as with the \nbackpropagation. and in addition there are direct connections from input to output cells. \nSee Figure 2. Each connection has an integer weight Wij that roughly corresponds to the \ninfluence of cell Uj on cell Ui . Although the weights of connections from the input layer \nto the intermediate layer are generated by using a random number generator (in this \npaper. integers between -10 and +10 were used) and fixed for learning process. Cell \nactivations are discrete. each taking on values +1. O. or-I. \n\n<Automated extraction of fuzzy IF-THEN rules> \n\n.\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7111 \n\u2022 \n\u2022 \u2022 \n'\" \u2022 \n\u2022 Knowledge \n\u2022 \n\u2022 \nInter- ~ ................. ~ \n\nEXlr.cllon \n[,of rul .. KnONledge \nanalysis \ni' \nengine \n\nbase \n\nInpUI of \n\nar.lnlna d.l, \n\nEditing .nd \u2022 \n.odlflc.llon \nof .xlr.cleCl User \nrul \u2022\u2022 \n\nface \n\n\u2022 \u2022 \u2022 \u2022 \u2022 \u2022 \u2022 \u2022 \u2022 \u2022 \n\nEnd \nusers \n\n,/ \n\n\"' R \u2022\u2022 ull. of \n\n........ \n\nConnr \u2022\u2022 llon of \n\nInpul d... \n\nTr.lnlna \nd.l .... \n\n1/ \n'A \u2022\u2022 III I \u2022 \n\nIfjeger wel\",t \n\nMatrix controlled ~ matrtxof \nInference engine \n\nneural networ. \n\n~ \n\n\u2022 <Embodlement. generalization and synthesis \n\u2022 \n\n~ .................................\u2022 \n\nof Informatlonby learning> \n\nA proposed fuzzy neural network \n\nFigure 1: A Schematic Diagram of A Fuzzy Neural System with \n\nAutomated Extraction of Fuzzy IF-THEN Rules \n\nActivations of the input cells Ii (i = 1.2 ..... p). the intermediate cell Hj (j = 1.2 ..... q) and \nthe output cell Ok (k = 1.2 ..... r) can be calculated using equations (1) - (4). The value of \nthe cell 10 is always + 1. and it is connected to every other cell except for input cells. \nSH\u00b7 = L w\u00b7-/-\nJI I \n\nSDk = LUkiIi + L VkjHj \n\n(1) \n\n(3) \n\np \n\nJ \n\np \n. 0 \n1= \n\nq \n. \n)=1 \n\ni=O \n+1 or True \no or Unknown (SHj = 0 ) (2) \n-lor False \n\n(SHj > 0 ) \n\n(SHj < 0 ) \n\n+1 or True \n\n(SOk > 0) \n\nOk = 0 or Unknown (SOk = 0) (4) \n\n{\n\n-lor False \n\n(SOk < 0) \n\n\f580 \n\nHayashi \n\nIntermediate \n\nLayer \n\nBias Cell \n\n----I .. ~ Trainable Connections \n\n---~ .. _ Randomly Generated Connections \n\nFigure 2: A Distributed Neural Network \n\n2.2 Fuzzy Neural Network \nTo handle various fuzziness in the input layer of the distributed neural network, it is \nnecessary to interpret subjective input data which has non-Boolean quantitative and/or \nqualitative meaning. In general, fuzzy sets defmed by monotone membership functions \ncan be \"defuzzified\" into a family of crisp sets by using the level set representation \n(Negoita, 1985) or \"thermometer code\" of B. Widrow. Therefore, the fuzziness can be \nincorporated into the training data by using only Boolean inputs. Once the training data \nis set up in this manner, it can be processed by the Pocket Algorithm (Gallant, 1990). In \nthis paper, we will propose aJuzzy neural network to handle fuzzy data and crisp data \n\nI@ @ \n\n@ \n\nOutput Layer \n\n1 \n.... \n\n\u00ae\u00ae \n/ \n\nBIas Cell \n\nintermediate Layer \n\n\u00aeI \n\\ \n\n'---v---' ... '---v-----\" oo\u00b7\u00b7\u00b7ooo\u00b7\u00b7@ \n~ .. @ 00\u00b7\u00b7\u00b70 \n\n'---v---' ... '---v---' \n\nGf \nG! \nFuzzy Cell Group \n\nGf \nGg \nCrisp Cell Group \n\nInput Layer \n\nFigure 3: A Neural Network with Fuzzy Cell Groups and Crisp Cell Groups \n\ngiven in the input layer. Figure 3 shows the structure of proposed fuzzy neural network \nwhose input layer consists of fuzzy cell groups and crisp (non-fuzzy) cell groups. Here, \n\n\fA Neural Expert System with Automated Extraction of fuzzy If-Then Rules \n\n581 \n\ntruthfulness of fuzzy information and crisp information such as binary encoded data is \nrepresented by fuzzy cell groups and crisp cell groups. respectively. A fuzzy cell group \nconsists of m input cells which have the level set representation using binary m(cid:173)\ndimensional vector. each taking on values in (+1. -I); whereas a crisp cell group also \nconsists of m input cells which take on two values in {(+ 1.+ 1 \u2022...\u2022 + 1). (-I.-I \u2022...\u2022 -I)}. \n\n3 AUTOMATED EXTRACTION OF FUZZY IF-THEN RULES FROM \nTRAINED NEURAL NETWORKS \n\nThis paper also extends previous work described in (Hayashi & Nakai. 1990) and \nproposes a method to extract automatically fuzzy If-Then rules with linguistic relative \nimportance of each proposition in an antecedent (Hayashi & Nakai. 1989) from a trained \nfuzzy neural network. The method is implemented in the knowledge analysis engine in \nFigure 1. The linguistic relative importance such as Very Important and Moderately \nImportant. which is defined by a fuzzy set. represents the degree of effect of each \nproposition on consequence. By providing linguistic relative importance for each \nproposition. each fuzzy If-Then rule has more flexible expression than that of ordinary If(cid:173)\nThen rules. Furthermore. truthfulness of each fuzzy If-Then rule is given in the form of \nlinguistic truth value such as Very True and Possibly True. which is defined by a fuzzy \nset. Enhancement of the presentation capability and flexibility by using fuzzy If-Then \nrules with linguistic relative importance facilitates the automated extraction of fuzzy If(cid:173)\nThen rules from a trained neural network. \n\n3.1 Automated If-Then Rule Extraction Algorithm \nWe have proposed some methods to extract fuzzy If-Then rules with linguistic relative \nimportance from a trained (fuzzy) neural network. In this section, we extend work \nreported in (Hayashi & Nakai. 1990; Hayashi et al., 1990) and give an algorithm to \nextract fuzzy If-Then rules from a trained fuzzy neural network in the following. Note \nthat an exact algorithm of Step 2 and Step 3 can be derived from algorithms shown in \n(Hayashi & Nakai. 1990) in the same manner. Here. we will give a brief discussion on \nthem due to space limitation. We shall concentrate on Stepl. \nStep l. Extraction of framework of fuzzy If-then rules: We select \npropositions in an antecedent (If-part) of a rule, that is. extract framework of fuzzy If(cid:173)\nThen rules. We will give a precise algorithm for this step in section 3.2. \nStep 2. Assignment of linguistic truth value to each \nextracted rule: A \nlinguistic truth value such as Very Very True (V.V.T.) and Possibly True (P.T.) is given \nto each fuzzy If-Then rule selected in Step 1. Linguistic truth value assigned to each rule \nindicates the degree of certainty to draw the conclusion. The linguistic truth value is \ndetermined by the relative amount of weighted sum of output cells. \nStep 3. Assignment of \nto each \nproposition: Linguistic relative importance is assigned to each proposition of \nantecedent in fuzzy If-Then rules. Linguistic relative importance such as Very Important \n(V.I.) and Moderately Important (M.I.) represents the degree of effect of each proposition \non consequence. \n\nimportance \n\nlinguistic \n\nrelative \n\n3.2 Algorithm to extract framework of fuzzy If-Then rules \nExtraction of dispensable propositions on cell groups in an antecedent (I f -part) is \nrequired for the extraction of framework of fuzzy If-Then rules. For simplicity. it is \n\n\f582 \n\nHayashi \n\nsupposed in this section that each cell consists of three input cells. Therefore, a fuzzy \ncell group takes on three values in (+1.-1,-1), (+1,+1,-1), (+1,+1.+1)}; whereas a crisp \ncell group takes on two values in (+1,+1,+1), (-1,-1,-1)}. In distributed neural network, \nwe can determine activations (values) of cells using partial input information. For \nexample, activations of intermediate cell Hj are determined as \n\n+1 or True \n\nHj = 0 or Unknown \n\n( ISHjl> USHj and SHj > 0 ) \n( ISHjl~ USHj) \n(ISHjl> USHj and SHj < 0 ) \n\n-lor False \n\nwhere \n\nUSH\u00b7 = \n\nJ \n\nL IW\u00b7i l . \n. I . U-I-1... \nJ: i IS n.vwwn \n\nIn the same manner, activations of output cell Ok are determined as \n( ISOkl> USOk and SOk > 0 ) \n(ISOkl~ USOk) \n(ISOkl> USOk and SHk < 0 ) \n\n0 or Unknown \n\n{\nOk = \n\n+1 or True \n\n-lor False \n\nwhere \n\nUSOk = \n\nL IUki l + \n\ni : Ii is Unknown \n\nL IVkjl \n\nj : H j is UnknOwn \n\n. \n\n(5) \n\n(6) \n\n(7) \n\n(8) \n\nOur problem is to determine the value of input cell groups so that each output cell Ok \ntakes on values +1 or -1. Propositions (Input items) corresponding to determined input \ncell groups will be entrapped in an antecedent (If-part) of each rule. We will give an \nextraction algorithm for framework of fuzzy If-Then rules as follows: \nStep I: Select one output cell 0 k . \nStep II: Select one cell group. If the selected cell group is a fuzzy cell group, set the \nvalues of the cell group in (+1,-1,-1), (+1,+1,-1) or (+1,+1,+1); whereas if the selected \ncell group is a crisp cell group, set the values of the cell group in (+1,+1,+1) or (-1,-1,-\n1). Furthermore, set the value of cell groups which were not selected to (0,0,0). \nStep III (Forward search): Determine all the value of intermediate cells Hj by \nusing the values of cell groups given in Step II and equation (5). Furthermore, \ndetermine the value of output cell Ok using (7). If the value of Ok is + 1 or -1, go to \nStep V. Otherwise (the value of Ok is 0), go to Step IV. Although all the cell groups are \nentrapped in an antecedent (If-part), if the value of Ok is 0, there is no framework of \nfuzzy If-Then rules for the output cell Ok and go to Step VI. \nStep IV (Backward search): Let v* be the maximum value of IVkjl which is an \nabsolute value of the weight of the connections between the output cell Ok and the \nintermediate cell Hj whose activation value is O. Furthermore, let u * be the maximum \nvalue of IUkil which is an absolute value of the weight of the connections between the \noutput cell Ok and the input cell Ii whose value is O. If u* ~ v* or values of all the \nintermediate cells are determined, go to Step IV-I. Otherwise, go to Step IV -2. \nStep IV-l: For the input cell Ii which is incident to uki ( I uki I = u* ). if the input cell Ii \nis included in the fuzzy cell grouP. go to Step IV-I-F; whereas in the crisp cell group, go \nto Step IV-I-C. \n\n\fA Neural Expert System with Automated Extraction of fuzzy If-Then Rules \n\n583 \n\nStep IV-I-F: If SOk '?O. select one pattern of the fuzzy cell group which has the \nmaximum value of SOk among (+1.-1.-1). (+1.+1.-1) and (+1.+1.+1). Conversely. If \nSOk < O. select one pattern which has the minimum value of SOk. Go to Step V. \nStep IV-I-C: If SOk ~O. select one pattern of the crisp cell group which has the \nmaximum value of SOk in (+1.+1.+1) and (-1.-1.-1). Conversely. If SOk < O. select one \npattern which has the minimum value of SOk. Go to Step V. \nStep IV-2: Let w* be he maximum value of I Wji I which is an absolute value of the \nweight of the connections between the intermediate cell Hj which is incident to vkj ( I vkj I \n= v* ) and the input cell Ii whose activation value is O. Select the input cell Ii which is \nincident to the connection Wji ( I Wji I = w*). If the input cell Ii is included in the fuzzy \ncell grouP. go to Step IV -2-F; whereas in the crisp cell grouP. go to Step IV -2-C. \nStep IV-2-F: If SHj ~O. select one pattern of the fuzzy cell group which has the \nmaximum value of SHj among (+1.-1.-1). (+1.+1.-1) and (+1.+1.+1). Conversely. If SHj \n< O. select one pattern which has the minimum value of SHj. Go to Step V. \nStep IV-2-C: If SHj '?O. select one pattern of the crisp cell group which has the \nmaximum value of SHj in (+1.+1.+1) and (-1.-1.-1). Conversely. If SHj < O. select one \npattern which has the minimum value of SUj- Go to Step V. \nStep V (Extraction or framework or If-then Rules): If the value of 0 k \nis \ndetermined. extract input items corresponding to a determined cell group as the \npropositions in an antecedent (I f -part). Here. if the value of Ok is + 1. the consequence \nis set to \"Ok is True\"; conversely if the value of Ok is -1. the consequence is set to \"Ok \nis False\". If multiple frameworks of If-Then rules with same antecedent and consequence \nare extracted. adopt one of them. \nStep VI (Termination condition or extraction algorithm for each output \ncell): For output cell Ok. if there are any cell groups which are not selected yet; or for \nselected cell groups. there are any patterns which are not selected yet. go to Step II. \nOtherwise. go to Step VII. \nStep VII (Termination condition or whole extraction algorithm): Repeat \nSteps II through VI stated above until the termination condition of extraction algorithm \nfor each output cell is satisfied. If there are any output cell Ok which are not selected yet \nin Step I. go to Step I. Otherwise. stop the whole extraction algorithm. \n\n4 APPLICATION TO MEDICAL DIAGNOSIS \n\nTo prove the effectiveness and validity of the proposed neural expert system. we have \ndeveloped neural expert systems for diagnosing hepatobiliary disorders (Yoshida et al .\u2022 \n1989 & 1990). We used a real medical database containing sex and the results of nine \nbiochemical tests (e.g. GOT. GGT) of four hepatobiliary disorders. Alcoholic liver \ndamage. Primary hepatoma. Liver cirrhosis and Cholelithiasis. The subjects consisted of \n536 patients who were admitted to a university-affiliated hospital. The patients were \nclinically and pathologically diagnosed by physicians. The subjects were randomly \nassigned to 373 training data and 163 test (external) data. Degree of abnormality of each \nbiochemical item is represented by a fuzzy cell group which consists of three input cells. \nThere are four output cells. Each output cell corresponds to a hepatobiliary disorder. Fifty \nthousand iterations in learning process of Pocket Algorithm was performed for each \n\n\f584 \n\nHayashi \n\noutput cell. The diagnosis criteria is the same as that employed in (Yoshida et a1. 1989). \nAfter learning by using training data from 345 patients, the fuzzy neural network \ncorrectly diagnosed 75.5% of test (external) data from 163 previously unseen patients and \ncorrectly diagnosed 100% of the training data. Conversely, the diagnostic accuracy of the \nlinear discriminant analysis was 65.0% of the test data and 68.4% of the training data. \nThe proposed fuzzy neural network showed significantly higher diagnostic accuracy in \ntraining data and also had substantially higher diagnostic accuracy in test data than those \nof linear discriminant analysis. We extracted 48 general fuzzy If-Then rules for diagnosing \nhepatobiliary disorders by using the proposed algorithm given in section 3.2. The number \nof rules for comfrrming diseases are 12 and the those for excluding diseases are 36. \nHayashi and Nakai (1989) have proposed three kinds of reasoning methods using fuzzy If(cid:173)\nThen rules with linguistic relative importance. In the present paper, we use the reasoning \nmethod-I for the evaluation of extracted fuzzy If-Then rules. Total diagnostic accuracy of \nthe twelve extracted rules (four confmning rules and eight excluding rules) is 87.7%. We \nconclude that the present neural network knowledge base approach will be a promising \nand useful technique for generating practical knowledge bases from various databases. It \nshould be noted that enhancement of interpretation capability of real data, and \nembodiment of implicit and/or subjective knowledge will lead to significant reduction of \nman power for knowledge acquisition in expert system development \n\nAcknowledgements \nThe author wishes to thank Dr. Stephen I. Gallant, Dr. Katsumi Yoshida and Mr. \nAtsushi Imura for their valuable comments and discussions. \n\nReferences \nGallant, S.I. 1988 Connectionist Expert Systems, CACM, 31(2), 152-169 \nGallant, S.I. & Hayashi, Y. 1990 A Neural Network Expert System with Confidence \nMeasurements, Proc. of the Third Int. Conf. on Infor. Proc. and Mgt. of Uncertainty in \nKnowledge-based Systems, pp.3-5, Paris, July 2-6; Springer Edited Volume (in press) \nGallant, S.I. 1990 Perceptron-Based Learning Algorithms, IEEE Transactions on \nNeural Networlcs, 1(2), 179-191 \nHayashi, Y. & Nakai, M. 1989 Reasoning Methods Using a Fuzzy Production Rule \nwith Linguistic Relative Importance in an Antecedent, The Transactions of The Institute \nof Electrical Engineers of Japan (T. lEE Japan), I09-C(9), 661-668 \nHayashi, Y. & Nakai, M. 1990 Automated Extraction of Fuzzy IF-THEN Rules Using \nNeural Networks, T. lEE Japan, llO-C(3), 198-206 \nHayashi, Y., Imura, A. & Yoshida, K. 1990 A Neural Expert System under Uncertain \nEnvironments and Its Evaluation, Proc. of the 11th Knowledge and Intelligence System \nSymposium, pp.13-18, Tokyo \nNegoita, C.V. 1985 Expert Systems and Fuzzy Systems: Benjamin Cummings Pub. \nYoshida, K., Hayashi, Y. & Imura, A. 1989 A Connectionist Expert System for \nDiagnosing Hepatobiliary Disorders,\" in MEDINF089 (proc. of the Sixth Conf. on \nMedical Informatics), B. Barber et al. eds.: North-Holland, 116-120 \nYoshida, K., Hayashi, Y., Imura, A. & Shimada, N. 1990 Fuzzy Neural Expert \nSystem for Diagnosing HepatobiIiary Disorders, Proc. of the Int. Conf. on Fuzzy Logic \n& Neural Networks (lIZUKA '90), pp.539-543, lizuka, Japan, July 20-24 \n\n\f", "award": [], "sourceid": 355, "authors": [{"given_name": "Yoichi", "family_name": "Hayashi", "institution": null}]}