`
k_lb
  • 浏览: 802217 次
  • 性别: Icon_minigender_1
  • 来自: 郑州
社区版块
存档分类
最新评论
  • kitleer： 据我所知，国内有款ETL调度监控工具TaskCTL，支持ket ...
    kettle调度

CentOS 安装 hadoop(伪分布模式)

阅读更多
在本机上装的CentOS 5.5 虚拟机，

软件准备：jdk 1.6 U26

hadoop:hadoop-0.20.203.tar.gz


ssh检查配置

Linux代码 收藏代码
  1. [root@localhost~]#ssh-keygen-trsa
  2. Generatingpublic/privatersakeypair.
  3. Enterfileinwhichtosavethekey(/root/.ssh/id_rsa):
  4. Createddirectory'/root/.ssh'.
  5. Enterpassphrase(emptyfornopassphrase):
  6. Entersamepassphraseagain:
  7. Youridentificationhasbeensavedin/root/.ssh/id_rsa.
  8. Yourpublickeyhasbeensavedin/root/.ssh/id_rsa.pub.
  9. Thekeyfingerprintis:
  10. a8:7a:3e:f6:92:85:b8:c7:be:d9:0e:45:9c:d1:36:3broot@localhost.localdomain
  11. [root@localhost~]#
  12. [root@localhost~]#cd..
  13. [root@localhost/]#cdroot
  14. [root@localhost~]#ls
  15. anaconda-ks.cfgDesktopinstall.loginstall.log.syslog
  16. [root@localhost~]#cd.ssh
  17. [root@localhost.ssh]#catid_rsa.pub>authorized_keys
  18. [root@localhost.ssh]#
  20. [root@localhost.ssh]#sshlocalhost
  21. Theauthenticityofhost'localhost(127.0.0.1)'can'tbeestablished.
  22. RSAkeyfingerprintis41:c8:d4:e4:60:71:6f:6a:33:6a:25:27:62:9b:e3:90.
  23. Areyousureyouwanttocontinueconnecting(yes/no)?yes
  24. Warning:Permanentlyadded'localhost'(RSA)tothelistofknownhosts.
  25. Lastlogin:TueJun2122:40:312011
  26. [root@localhost~]#

安装jdk

Linux代码 收藏代码
  1. [root@localhostjava]#chmod+xjdk-6u26-linux-i586.bin
  2. [root@localhostjava]#./jdk-6u26-linux-i586.bin
  3. ......
  4. ......
  5. ......
  6. FormoreinformationonwhatdataRegistrationcollectsand
  7. howitismanagedandused,see:
  8. http://java.sun.com/javase/registration/JDKRegistrationPrivacy.html
  10. PressEntertocontinue.....
  13. Done.

安装完成后生成文件夹：jdk1.6.0_26

配置环境变量

Linux代码 收藏代码
  1. [root@localhostjava]#vi/etc/profile
  2. #添加如下信息
  3. #setjavaenvironment
  4. exportJAVA_HOME=/usr/java/jdk1.6.0_26
  5. exportCLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
  6. exportPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/bin:$PATH:$HOME/bin
  7. exportHADOOP_HOME=/usr/local/hadoop/hadoop-0.20.203
  8. exportPATH=$PATH:$HADOOP_HOME/bin
  10. [root@localhostjava]#chmod+x/etc/profile
  11. [root@localhostjava]#source/etc/profile
  12. [root@localhostjava]#
  13. [root@localhostjava]#java-version
  14. javaversion"1.6.0_26"
  15. Java(TM)SERuntimeEnvironment(build1.6.0_26-b03)
  16. JavaHotSpot(TM)ClientVM(build20.1-b02,mixedmode,sharing)
  17. [root@localhostjava]#

修改hosts

Linuxa代码 收藏代码
  1. [root@localhostconf]#vi/etc/hosts
  2. #Donotremovethefollowingline,orvariousprograms
  3. #thatrequirenetworkfunctionalitywillfail.
  4. 127.0.0.1localhost.localdomainlocalhost
  5. ::1localhost6.localdomain6localhost6
  6. 127.0.0.1namenodedatanode01

解压安装hadoop

Linux代码 收藏代码
  1. [root@localhosthadoop]#tarzxvfhadoop-0.20.203.tar.gz
  2. ......
  3. ......
  4. ......
  5. hadoop-0.20.203.0/src/contrib/ec2/bin/image/create-hadoop-image-remote
  6. hadoop-0.20.203.0/src/contrib/ec2/bin/image/ec2-run-user-data
  7. hadoop-0.20.203.0/src/contrib/ec2/bin/launch-hadoop-cluster
  8. hadoop-0.20.203.0/src/contrib/ec2/bin/launch-hadoop-master
  9. hadoop-0.20.203.0/src/contrib/ec2/bin/launch-hadoop-slaves
  10. hadoop-0.20.203.0/src/contrib/ec2/bin/list-hadoop-clusters
  11. hadoop-0.20.203.0/src/contrib/ec2/bin/terminate-hadoop-cluster
  12. [root@localhosthadoop]#

进入hadoop配置conf

Linux代码 收藏代码
  1. ####################################
  2. [root@localhostconf]#vihadoop-env.sh
  3. #添加代码
  4. #setjavaenvironment
  5. exportJAVA_HOME=/usr/java/jdk1.6.0_26
  7. #####################################
  8. [root@localhostconf]#vicore-site.xml
  9. <?xmlversion="1.0"?>
  10. <?xml-stylesheettype="text/xsl"href="configuration.xsl"?>
  11. <!--Putsite-specificpropertyoverridesinthisfile.-->
  12. <configuration>
  13. <property>
  14. <name>fs.default.name</name>
  15. <value>hdfs://namenode:9000/</value>
  16. </property>
  17. <property>
  18. <name>hadoop.tmp.dir</name>
  19. <value>/usr/local/hadoop/hadooptmp</value>
  20. </property>
  21. </configuration>
  23. #######################################
  24. [root@localhostconf]#vihdfs-site.xml
  25. <?xmlversion="1.0"?>
  26. <?xml-stylesheettype="text/xsl"href="configuration.xsl"?>
  27. <!--Putsite-specificpropertyoverridesinthisfile.-->
  28. <configuration>
  29. <property>
  30. <name>dfs.name.dir</name>
  31. <value>/usr/local/hadoop/hdfs/name</value>
  32. </property>
  33. <property>
  34. <name>dfs.data.dir</name>
  35. <value>/usr/local/hadoop/hdfs/data</value>
  36. </property>
  37. <property>
  38. <name>dfs.replication</name>
  39. <value>1</value>
  40. </property>
  41. </configuration>
  43. #########################################
  44. [root@localhostconf]#vimapred-site.xml
  45. <?xmlversion="1.0"?>
  46. <?xml-stylesheettype="text/xsl"href="configuration.xsl"?>
  47. <!--Putsite-specificpropertyoverridesinthisfile.-->
  48. <configuration>
  49. <property>
  50. <name>mapred.job.tracker</name>
  51. <value>namenode:9001</value>
  52. </property>
  53. <property>
  54. <name>mapred.local.dir</name>
  55. <value>/usr/local/hadoop/mapred/local</value>
  56. </property>
  57. <property>
  58. <name>mapred.system.dir</name>
  59. <value>/tmp/hadoop/mapred/system</value>
  60. </property>
  61. </configuration>
  63. #########################################
  64. [root@localhostconf]#vimasters
  65. #localhost
  66. namenode
  68. #########################################
  69. [root@localhostconf]#vislaves
  70. #localhost
  71. datanode01

启动 hadoop

Linux代码 收藏代码
  1. #####################<spanstyle="font-size:small;">格式化namenode##############</span>
  5. [root@localhostbin]#hadoopnamenode-format
  6. 11/06/2300:43:54INFOnamenode.NameNode:STARTUP_MSG:
  7. /************************************************************
  8. STARTUP_MSG:StartingNameNode
  9. STARTUP_MSG:host=localhost.localdomain/127.0.0.1
  10. STARTUP_MSG:args=[-format]
  11. STARTUP_MSG:version=0.20.203.0
  12. STARTUP_MSG:build=http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203-r1099333;compiledby'oom'onWedMay407:57:50PDT2011
  13. ************************************************************/
  14. 11/06/2300:43:55INFOutil.GSet:VMtype=32-bit
  15. 11/06/2300:43:55INFOutil.GSet:2%maxmemory=19.33375MB
  16. 11/06/2300:43:55INFOutil.GSet:capacity=2^22=4194304entries
  17. 11/06/2300:43:55INFOutil.GSet:recommended=4194304,actual=4194304
  18. 11/06/2300:43:56INFOnamenode.FSNamesystem:fsOwner=root
  19. 11/06/2300:43:56INFOnamenode.FSNamesystem:supergroup=supergroup
  20. 11/06/2300:43:56INFOnamenode.FSNamesystem:isPermissionEnabled=true
  21. 11/06/2300:43:56INFOnamenode.FSNamesystem:dfs.block.invalidate.limit=100
  22. 11/06/2300:43:56INFOnamenode.FSNamesystem:isAccessTokenEnabled=falseaccessKeyUpdateInterval=0min(s),accessTokenLifetime=0min(s)
  23. 11/06/2300:43:56INFOnamenode.NameNode:Cachingfilenamesoccuringmorethan10times
  24. 11/06/2300:43:57INFOcommon.Storage:Imagefileofsize110savedin0seconds.
  25. 11/06/2300:43:57INFOcommon.Storage:Storagedirectory/usr/local/hadoop/hdfs/namehasbeensuccessfullyformatted.
  26. 11/06/2300:43:57INFOnamenode.NameNode:SHUTDOWN_MSG:
  27. /************************************************************
  28. SHUTDOWN_MSG:ShuttingdownNameNodeatlocalhost.localdomain/127.0.0.1
  29. ************************************************************/
  30. [root@localhostbin]#
  32. ###########################################
  33. [root@localhostbin]#./start-all.sh
  34. startingnamenode,loggingto/usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-namenode-localhost.localdomain.out
  35. datanode01:startingdatanode,loggingto/usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-datanode-localhost.localdomain.out
  36. namenode:startingsecondarynamenode,loggingto/usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-secondarynamenode-localhost.localdomain.out
  37. startingjobtracker,loggingto/usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-jobtracker-localhost.localdomain.out
  38. datanode01:startingtasktracker,loggingto/usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-tasktracker-localhost.localdomain.out
  39. [root@localhostbin]#jps
  40. 11971TaskTracker
  41. 11807SecondaryNameNode
  42. 11599NameNode
  43. 12022Jps
  44. 11710DataNode
  45. 11877JobTracker

查看集群状态

Linux代码 收藏代码
  1. [root@localhostbin]#hadoopdfsadmin-report
  2. ConfiguredCapacity:4055396352(3.78GB)
  3. PresentCapacity:464142351(442.64MB)
  4. DFSRemaining:464089088(442.59MB)
  5. DFSUsed:53263(52.01KB)
  6. DFSUsed%:0.01%
  7. Underreplicatedblocks:0
  8. Blockswithcorruptreplicas:0
  9. Missingblocks:0
  11. -------------------------------------------------
  12. Datanodesavailable:1(1total,0dead)
  14. Name:127.0.0.1:50010
  15. DecommissionStatus:Normal
  16. ConfiguredCapacity:4055396352(3.78GB)
  17. DFSUsed:53263(52.01KB)
  18. NonDFSUsed:3591254001(3.34GB)
  19. DFSRemaining:464089088(442.59MB)
  20. DFSUsed%:0%
  21. DFSRemaining%:11.44%
  22. Lastcontact:ThuJun2301:11:15PDT2011
  25. [root@localhostbin]#

其他问题：1

Linux代码 收藏代码
  1. ####################启动报错##########
  2. [root@localhostbin]#./start-all.sh
  3. startingnamenode,loggingto/usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-namenode-localhost.localdomain.out
  4. Theauthenticityofhost'datanode01(127.0.0.1)'can'tbeestablished.
  5. RSAkeyfingerprintis41:c8:d4:e4:60:71:6f:6a:33:6a:25:27:62:9b:e3:90.
  6. Areyousureyouwanttocontinueconnecting(yes/no)?y
  7. Pleasetype'yes'or'no':yes
  8. datanode01:Warning:Permanentlyadded'datanode01'(RSA)tothelistofknownhosts.
  9. datanode01:startingdatanode,loggingto/usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-datanode-localhost.localdomain.out
  10. <strong><spanstyle="color:#ff0000;">datanode01:Unrecognizedoption:-jvm
  11. datanode01:CouldnotcreatetheJavavirtualmachine.</span>
  14. </strong>
  18. namenode:startingsecondarynamenode,loggingto/usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-secondarynamenode-localhost.localdomain.out
  19. startingjobtracker,loggingto/usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-jobtracker-localhost.localdomain.out
  20. datanode01:startingtasktracker,loggingto/usr/local/hadoop/hadoop-0.20.203/bin/../logs/hadoop-root-tasktracker-localhost.localdomain.out
  21. [root@localhostbin]#jps
  22. 10442JobTracker
  23. 10533TaskTracker
  24. 10386SecondaryNameNode
  25. 10201NameNode
  26. 10658Jps
  28. ################################################
  29. [root@localhostbin]#vihadoop
  30. elif["$COMMAND"="datanode"];then
  31. CLASS='org.apache.hadoop.hdfs.server.datanode.DataNode'
  32. if[[$EUID-eq0]];then
  33. HADOOP_OPTS="$HADOOP_OPTS-jvmserver$HADOOP_DATANODE_OPTS"
  34. else
  35. HADOOP_OPTS="$HADOOP_OPTS-server$HADOOP_DATANODE_OPTS"
  36. fi
  38. #http://javoft.net/2011/06/hadoop-unrecognized-option-jvm-could-not-create-the-java-virtual-machine/
  39. #改为
  40. elif["$COMMAND"="datanode"];then
  41. CLASS='org.apache.hadoop.hdfs.server.datanode.DataNode'
  42. #if[[$EUID-eq0]];then
  43. #HADOOP_OPTS="$HADOOP_OPTS-jvmserver$HADOOP_DATANODE_OPTS"
  44. #else
  45. HADOOP_OPTS="$HADOOP_OPTS-server$HADOOP_DATANODE_OPTS"
  46. #fi
  48. #或者换非root用户启动
  49. #启动成功

2，启动时要关闭防火墙

查看运行情况：

http://localhost:50070

Firefox显示代码 收藏代码
  1. NameNode'localhost.localdomain:9000'
  2. Started:ThuJun2301:07:18PDT2011
  3. Version:0.20.203.0,r1099333
  4. Compiled:WedMay407:57:50PDT2011byoom
  5. Upgrades:Therearenoupgradesinprogress.
  7. Browsethefilesystem
  8. NamenodeLogs
  9. ClusterSummary
  10. 6filesanddirectories,1blocks=7total.HeapSizeis31.38MB/966.69MB(3%)
  11. ConfiguredCapacity:3.78GB
  12. DFSUsed:52.01KB
  13. NonDFSUsed:3.34GB
  14. DFSRemaining:442.38MB
  15. DFSUsed%:0%
  16. DFSRemaining%:11.44%
  17. LiveNodes:1
  18. DeadNodes:0
  19. DecommissioningNodes:0
  20. NumberofUnder-ReplicatedBlocks:0
  22. NameNodeStorage:
  23. StorageDirectoryTypeState
  24. /usr/local/hadoop/hdfs/nameIMAGE_AND_EDITSActive

http://localhost:50030

Firefox显示代码 收藏代码
  1. namenodeHadoopMap/ReduceAdministration
  2. QuickLinks
  4. *SchedulingInfo
  5. *RunningJobs
  6. *RetiredJobs
  7. *LocalLogs
  9. State:RUNNING
  10. Started:ThuJun2301:07:30PDT2011
  11. Version:0.20.203.0,r1099333
  12. Compiled:WedMay407:57:50PDT2011byoom
  13. Identifier:201106230107
  14. ClusterSummary(HeapSizeis15.31MB/966.69MB)
  15. RunningMapTasksRunningReduceTasksTotalSubmissionsNodesOccupiedMapSlotsOccupiedReduceSlotsReservedMapSlotsReservedReduceSlotsMapTaskCapacityReduceTaskCapacityAvg.Tasks/NodeBlacklistedNodesGraylistedNodesExcludedNodes
  16. 00010000224.00000
  18. SchedulingInformation
  19. QueueNameStateSchedulingInformation
  20. defaultrunningN/A
  21. Filter(Jobid,Priority,User,Name)
  22. Example:'user:smith3200'willfilterby'smith'onlyintheuserfieldand'3200'inallfields
  23. RunningJobs
  24. none
  25. RetiredJobs
  26. none
  27. LocalLogs
  28. Logdirectory,JobTrackerHistoryThisisApacheHadooprelease0.20.203.0

测试：

Linux代码 收藏代码
  1. ##########建立目录名称##########
  2. [root@localhostbin]#hadoopfs-mkdirtestFolder
  4. ###############拷贝文件到文件夹中
  5. [root@localhostlocal]#ls
  6. binetcgameshadoopincludeliblibexecsbinsharesrcSSH_key_file
  7. [root@localhostlocal]#hadoopfs-copyFromLocalSSH_key_filetestFolder
  9. 进入web页面即可查看

参考：http://bxyzzy.blog.51cto.com/854497/352692

附： 准备FTP :yum install vsftpd (方便文件传输 和hadoop无关)

关闭防火墙：service iptables start

启动FTP：service vsftpd start


分享到：
| CentOs vsftp
评论
发表评论

您还没有登录,请您登录后再发表评论

相关推荐

Magicbox
Global site tag (gtag.js) - Google Analytics