Click here to Skip to main content
15,867,308 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I have setup a single node hadoop cluster (2.71.1) on windows 7 and now trying to establish it's connection with Azure storage (wasb) with no success.

I am getting the error:

No FileSystem for scheme: wasb

I have been following several blogs but was focused on : articles/hadoopAndWasb.md at master · hning86/articles · GitHub[^]

What I have tried:

My core-site.xml looks like this:
<configuration>

<property>
<name>fs.default.name
<value>hdfs://localhost:50000


<property>
<name>fs.AbstractFileSystem.wasb.Impl
<value>org.apache.hadoop.fs.azure.Wasb


<property>
<name>fs.azure.account.key.My_Account_Name.blob.core.windows.net

<value>My_Key



I have added the jars to HADOOP_CLASSPATH var in hadoop-env.cmd as below

set HADOOP_CLASSPATH=C:/Users/rj145/Desktop/hadoop2.7/hadoop-2.7.0/share/hadoop/tools/lib/hadoop-azure-2.7.0.jar:C:/Users/rj145/Desktop/hadoop2.7/hadoop-2.7.0/share/hadoop/tools/lib/azure-storage-2.0.0.jar:$HADOOP_CLASSPATH
Posted
Updated 15-Sep-17 20:35pm
Comments
[no name] 18-Aug-22 6:57am    
Our online selenium testing course can help you become an expert in architecture and selenium components. You can master the whole 3.0 suite by taking the greatest selenium course available online.

First you need to follow some steps for enable WASB on Hadoop

•	We need to create an account on windows azure.
•	Than take service 
•	Than we need to implement Hadoop.

Follow this to better understanding:
https://blogs.msdn.microsoft.com/cindygross/2015/02/04/understanding-wasb-and-hadoop-storage-in-azure/
 
Share this answer
 
v3
hi
here is the step by step process

[^]WASB is automatically enabled in HDInsight clusters. But you can also mount a blob storage account manually to a [^]Hadoop Administration instance that lives anywhere as long as it has Internet access to the blob storage. Here are the steps:

I assume that you have a standard Apache distrition of Hadoop 2.7.1 installed in /usr/local/hadoop directory on a Linux box.

#Step 1. Verify your Hadoop version This needs to be 2.7.1 or later.

/usr/local/hadoop/bin/hadoop version

#Step 2. Verify that you have these 2 jar files in your hadoop installation.

ls /usr/local/hadoop/share/hadoop/tools/lib/hadoop-azure-2.7.1.jar
ls /usr/local/hadoop/share/hadoop/tools/lib/azure-storage-2.0.0.jar

#Step 3. Modify hadoop-env.sh Modify $HADOOP_HOME/etc/hadoop/hadoop-env.sh file to add these 2 jar files to Hadoop classpath at the end of the file.

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/hadoop/share/hadoop/tools/lib/hadoop-azure-2.7.1.jar:/usr/local/hadoop/share/hadoop/tools/lib/azure-storage-2.0.0.jar

#Step 4. Modify core-site.xml Modify $HADOOP_HOME/etc/hadoop/core-site.xml file to add these name-value pairs in the configuration to point to the blob storage.

<property>
<name>fs.AbstractFileSystem.wasb.Impl</name>
<value>org.apache.hadoop.fs.azure.Wasb</value>
</property>

<property>
<name>fs.azure.account.key.my_blob_account_name.blob.core.windows.net</name>
<value>my_blob_account_key</value>
</property>

<!-- optionally set the default file system to a container -->
<property>
<name>fs.defaultFS</name>
<value>wasb://my_container_name@my_blob_account_name.blob.core.windows.net</value>
</property>
If you don't want to expose your storage account key in core-site.xml. You can also encrypt it. Read more here.

#Step 5. Test the connection.

/usr/local/hadoop/bin/hadoop fs -ls wasb:///
/usr/local/hadoop/bin/hadoop fs -ls wasb://my_container_name@my_blob_account_name.blob.core.windows.net
/usr/local/hadoop/bin/hadoop fs -ls / # if the default file system is set to wasb
 
Share this answer
 
v2

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900