学校代码： 10128 学 号： 040201015
本科毕业设计说明书
题 目：搜索引擎的设计与实现 学生姓名：庞佳
学 院：信息工程学院 系 别：计算机
专 业：计算机科学与技术 班 级：计算机04-2 指导教师：苏依拉 副教授
钱
庭荣 工程师
二 〇 〇 八 年 六 月
内蒙古工业大学本科毕业设计说明书
摘 要
为了适应网络信息的飞速增长，并且能够迅速、方便地从网络中获取有效信息, 搜索引擎逐渐走进了人们的生活，“竹竹”搜索引擎系统在这样的条件下，应运而生。
本文首先系统的介绍了搜索引擎的概念、发展历史、和搜索引擎的分类。使读者能够初步了解搜索引擎技术。然后，详细介绍了“竹竹”搜索引擎系统。
“竹竹”搜索引擎是基于Web的，面向笔记本电脑品牌的搜索引擎。系统的前端以MVC模式来实现，Spring做中间层，JDBC作后端来开发实现的。本系统分为三个子模块，抓取模块实现的功能为：将web上的海量网页抓取到系统中；采用的实现方法是使用Heritrix来完成对网页的抓取。处理模块实现的功能为：解析网页，提取其中的有用内容，为网页建立词库，由于笔记本电脑的品牌名在现有词库中不存在，因此要建立其特有的词库文件，对解析网页生成的信息文件进行分词，并建立索引，将索引存入数据库中；采用的实现方法是：通过Lucene的API来实现对网页内容的建索，使用HTMLParser的API实现了对网页内容的解析。用户模块实现的主要功能是：用户模块是系统的用户接口，用户通过此模块完成与系统的交互，当用户在查询界面上输入要检索的品牌信息后，系统将在可以接受的时间内，返回用户所需的结果集；采用的实现方法是：通过DWR封装了AJAX技术，处理用户请求；通过Lucene的API来实现检索。
关键词：搜索引擎；Lucene；Heritrix
内蒙古工业大学本科毕业设计说明书
Abstract
In order to adapt to the rapid growth of information networks, and can quickly and easily access to information from the network, search engines gradually come into people's lives, \
This paper first introduced the system,the concept of search engines, the development of history, and search engines category. So that readers can understand the search engine technology. Then, details of the \
\search engine is a Web-based, brand-oriented notebook computer search engine. The front-end system is made by model MVC, Spring to the middle layer, JDBC for the back-end . The system is divided into three sub-module, crawl module for the realization of the functions: Massive on the web page to crawl into the system; using the method is used to running Heritrix. Processing module for the realization of the functions: Analysis of the page, which extract useful content, pages thesaurus, because the brand of notebook computers available in the thesaurus does not exist, to establish its unique lexicon documents, analysis of the page Information generated by Word documents, and index, the index will be deposited in the database; method is used: Lucene API to achieve the content of the cable construction, the use of the API HTMLParser achieve the web content analysis. User module to achieve the main functions are: the user module is the user interface, the user through the completion of this module interactive system, when a user interface for input to the brand information retrieval system, the system will be acceptable time, Back to the user requirements set of results; using the method is: through the package the DWR AJAX technology, processing user requests through the Lucene API to achieve search.
Key words: search engine; Lucene; Heritrix
内蒙古工业大学本科毕业设计说明书
目 录
引 言 ................................................................................................................................... 1 第一章 课题背景 ................................................................................................................. 2
1.1搜索引擎的概念 ...................................................................................................... 2 1.2搜索引擎的发展历史 .............................................................................................. 3
1.2.1搜索引擎的起源 ............................................................................................ 3 1.2.2第一代搜索引擎 ............................................................................................ 3 1.2.3第二代搜索引擎 ............................................................................................ 3 1.2.4当前著名的搜索引擎简介 ............................................................................ 4 1.3搜索引擎的分类 ...................................................................................................... 5
1.3.1全文索引 ........................................................................................................ 5 1.3.2目录索引 ........................................................................................................ 5 1.3.3元搜索引擎 .................................................................................................... 5 1.3.4垂直搜索引擎 ................................................................................................ 6 1.3.5其他非主流搜索引擎形式 ............................................................................ 6
第二章 系统需求分析 ......................................................................................................... 7
2.1搜索引擎的工作原理 .............................................................................................. 7 2.2系统功能需求 .......................................................................................................... 7 2.3系统性能需求 .......................................................................................................... 8 第三章 系统总体设计 ......................................................................................................... 9
3.1“竹竹”搜索引擎系统总体介绍 ........................................................................... 9 3.2系统模块介绍 ........................................................................................................ 11
3.2.1 模块功能介绍 ............................................................................................. 11
第四章 系统详细设计 ....................................................................................................... 16
4.1模块总体介绍 ........................................................................................................ 16 4.2抓取子模块 ............................................................................................................ 17
4.2.1运行Heritrix子模块 ................................................................................... 17 4.2.2分析网页子模块 .......................................................................................... 22 4.3处理子模块 ............................................................................................................ 26
4.3.1解析网页子模块 .......................................................................................... 26
内蒙古工业大学本科毕业设计说明书
4.3.2创建词库子模块 .......................................................................................... 27 4.3.3生成持久化类子模块 .................................................................................. 27 4.3.4创建Document子模块 ............................................................................... 28 4.4.5存储数据子模块 .......................................................................................... 32 4.4用户子模块 ............................................................................................................ 32
4.4.1搜索页面 ...................................................................................................... 33 4.4.2详细信息页面 .............................................................................................. 33
结 论 ................................................................................................................................. 34 参考文献 ............................................................................................................................. 35 谢 辞 ................................................................................................................................. 36
搜索引擎的设计与实现
