A Versatile Crawler for Kademlia-Based Networks and Its Convergence Analysis

LIU Xiangtao; CAI Kai; YUE Qiang; JI Tongkai

Article Contents

Article Navigation > Chinese Journal of Electronics > 2013 > 22(1): 7-14

LIU Xiangtao, CAI Kai, YUE Qiang, et al., “A Versatile Crawler for Kademlia-Based Networks and Its Convergence Analysis,” Chinese Journal of Electronics, vol. 22, no. 1, pp. 7-14, 2013,

Citation:

LIU Xiangtao, CAI Kai, YUE Qiang, et al., “A Versatile Crawler for Kademlia-Based Networks and Its Convergence Analysis,” Chinese Journal of Electronics, vol. 22, no. 1, pp. 7-14, 2013,

LIU Xiangtao, CAI Kai, YUE Qiang, et al., “A Versatile Crawler for Kademlia-Based Networks and Its Convergence Analysis,” Chinese Journal of Electronics, vol. 22, no. 1, pp. 7-14, 2013,

Citation:

LIU Xiangtao, CAI Kai, YUE Qiang, et al., “A Versatile Crawler for Kademlia-Based Networks and Its Convergence Analysis,” Chinese Journal of Electronics, vol. 22, no. 1, pp. 7-14, 2013,

PDF( 615 KB)

A Versatile Crawler for Kademlia-Based Networks and Its Convergence Analysis

1.
Guangdong Electronics Industry Institute, Dongguan 523808, Guangdong, China;
2.
Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China

Funds: This work is supported by the National High Technology Research and Development Program of China (863 Program) (No.2011AA040506), Guangdong Province & Chinese Academy of Sciences Comprehensive Strategic Cooperation Project (No.2011A090100003) and Guangdong Talents Program.

Received Date: 2011-09-01
Rev Recd Date: 2012-03-01
Publish Date: 2013-01-05

Abstract

Abstract

In recent years, Peer-to-peer (P2P) file sharing networks such as BitTorrent and eMule become more and more popular. BitTorrent and eMule deploy their distributed networks based on Kademlia, a robust Distributed hash table (DHT) protocol to facilitate the delivery of contents. The Kademlia-based networks and its measurement tools (i.e., P2P crawlers) have intrigued many researchers in the P2P community. However, to our best knowledge, few versatile P2P crawlers are developed for intensive measurement on Kademlia-based networks. In this paper we develop such a crawler, namely Rainbow. For the first time, we theoretically analyze the convergence of Rainbow, and obtain its convergence order which determines the time complexity of crawling. We then experimentally verify the convergence of Rainbow. Finally, we demonstrate that Rainbow can be applied as a versatile measurement tool to identify various detailed characteristics for Kademlia-based networks.
- Peer-to-peer network,
- Kademlia,
- Measurement,
- Crawler,
- Rainbow,
- Convergence

FullText(HTML)

References(21)

References

P. Maymounkov, D. Mazieres, “Kademlia: a peer-to-peer informationsystem based on the XOR metric”, International Workshopon Peer-to-Peer Systems (IPTPS), Cambridge, MA, USA,pp.53-65, 2002.

J. Alderman, Sonic Boom: Napster, MP3, and the New Pioneersof Music, Perseus Pub., Michigan, USA, pp.89-108, 2001.

I. Stoica, R. Morris et al., “Chord: a scalable peer-to-peerlookup protocol for Internet applications”, IEEE/ACM Transactionson Networking, Vol.11, No.1, pp.17-32, 2003.

D. Stutzbach, R. Rejaie, “Understanding churn in peer-to-peernetworks”, Proceedings of the Internet Measurement Conference(IMC), Rio de Janeriro, Brazil, pp.189-202, 2006.

M. Steiner, T. En-Najjary, E. Biersack, “Long term study ofpeer behavior in the KAD DHT”, IEEE/ACM Transaction onNetworking, Vol.17, No.5, pp.1371-1384, 2009.

Y. Wang, R.C. Wang, Z.J. Han, “Dynamical trust constructionschema with fuzzy decision in P2P system”, Chinese Journal ofElectronics, Vol.18, No.3, pp.417-421, 2009.

X. Liu, X. Cheng, Y. Chen, S. Bai, “Improving the routingperformance of KAD through social network analysis”, IEEESymposium on Computers and Communications (ISCC), Riccione,Italy, pp.721-727, 2010.

Q. Deng, X. Lu, L. Chen, “Group clustering mechanism forP2P large scale data sharing collaboration”, Chinese Journalof Electronics, Vol.14, No.1, pp.58-61, 2005.

S. Naicken, A. Basu, B. Livingston, S. Rodhetbhai, “A surveyof peer-to-peer network simulators”, Proc. of the 7th AnnualPostgraduate Symposium (PGNet), Liverpool, UK, 2006.

I. Baumgart, B. Heep, S. Krause, “OverSim: A flexible overlaynetwork simulation framework”, IEEE Global Internet Symposium,Anchorage, AK, pp.79-84, 2007.

R. Bhagwan, S. Savage, G. Voelker, “Understanding availability”,Proc. of the 2nd International Workshop on Peer-to-PeerSystems (IPTPS), Berkeley, CA, USA, pp.256-267, 2003.

K. Kutzner, T. Fuhrmann, “Measuring large overlay networks— the Overnet example”, Proc. of the 14th KiVS, Kaiserslautern,Germany, pp.193-204, 2005.

J. Falkner, et al., “Profiling a million user DHT”, Proc. of theInternet Measurement Conference (IMC), New York, NY, USA,pp.129-134, 2007.

V. Sadafal, “Measurement and analysis of BitTorrent”, MasterThesis, Texas A&M University at Texas, USA, 2008.

D. Stutzbach, R. Rejaie, “Improving lookup performance overa widely-deployed DHT”, Proc. INFOCOM, Barcelona, Spain,pp.1-12, 2006.

M. Steiner, T. En-Najjary, E. Biersack, “A global view of KAD”,Proc. of the Internet Measurement Conference (IMC), SanDiego, CA, USA, pp.117-122, 2007.

M. Steiner, E. Biersack, T. En-Najjary, “Actively monitoringpeers in KAD”, Proc. of the 6th International Workshop onPeer-to-Peer Systems (IPTPS), Bellevue, WA, USA, 2007.

J. Yu, C. Fang, J. Xu, E.C. Chang, Z. Li, “ID repetition inKad”, Proc. of the 9th IEEE International Conference on Peerto-Peer Computing (P2P), Seattle, USA, pp.111-120, 2009.

T. Locher, et al., “A peer activity study in eDonkey & Kad”, InternationalWorkshop on Dynamic Networks: Algorithms andSecurity (DYNAS), Wroclaw, Poland, 2009.

M. Pietrzyk, et al., “Digging into KAD users’ shared folders”,SIGCOMM, Seattle, WA, USA, pp.493-494, 2008.

M. Iliofotou, et al., “Comparing BitTorrent clients in the wild:the case of download speed”, International Workshop on Peerto-Peer Systems (IPTPS), San Jose, CA, USA, 2010.

Relative Articles

Supplements(0)

Cited By

Proportional views