\nhttp:\/\/semantica.cs.lth.se\/pyspark\/#\/pyspark-vm<\/a><\/p>\n
Dla tych, kt\u00f3rzy na co dzie\u0144 korzystaj\u0105 ze stacji roboczej MS Windows.<\/p>\n
Potrzebne oprogramowanie b) Vagrant (nak\u0142adka na wirtualizator)<\/p>\n https:\/\/www.vagrantup.com\/downloads.html<\/a><\/p>\n Instalujemy obraz<\/p>\n To chwil\u0119 trwa, cierpliwo\u015bci, obraz\u00a0 ponad 1 GB obj\u0119to\u015bci.<\/p>\n Sprawdzamy zainstalowane obrazy maszyn wirtualnych.<\/p>\n Sprawdzamy status maszyn<\/p>\n Uruchamiamy maszyn\u0119 wirtualn\u0105<\/p>\n Jak wygl\u0105da plik VagrantFile, warto na niego spojrze\u0107, w szczeg\u00f3lno\u015bci na numery port\u00f3w, kt\u00f3re s\u0105 forwardowane.<\/p>\n Uwagi, kt\u00f3re mog\u0105 zmniejszy\u0107 frustracj\u0119:<\/strong><\/p>\n Je\u015bli zainstalujemy zbyt now\u0105 wersj\u0119 VirtualBoxa Vagrant jej nie wykryje, a co wi\u0119cej b\u0119dzie komunikowa\u0107 nas o braku HyperV.<\/p>\n Co ciekawe na komputerach z windows 7 pro\u00a0 instalacja tego typu oprogramowania jest mo\u017cliwa, ale tej wersji nie testowa\u0142em,<\/p>\n pozostajemy przy oprogramowaniu Virtualbox.<\/p>\n Dla dociekliwych:<\/p>\n https:\/\/www.altkomakademia.pl\/baza-wiedzy\/qna\/discussion\/2763\/instalacja-hyper-v-manager-w-windows-7-<\/a> Wersja PowerShella przed:<\/p>\n Po instalacji ze strony Microsoftu<\/p>\n https:\/\/www.microsoft.com\/en-us\/download\/details.aspx?id=40855<\/p>\n Wersja powershella po:<\/p>\n W wersji VirtualBox (5.1) domy\u015blnie nie ma dost\u0119pu do sieci zewn\u0119trznej od \u015brodka maszyny, przez co nie mo\u017cna zaktulizowa\u0107 jej oprogramowania. Pomaga poprawa w pliku konfiguracyjnym Vagrantfile<\/p>\n Nale\u017cy doda\u0107 w pliku Vagranfile dodatkowe linie kodu<\/p>\n Uruchamiany stron\u0119 http:\/\/localhost:8081<\/a><\/p>\n Jako silnik wybieramy pyspark<\/p>\n W pierwszej kom\u00f3rce wpisujemy<\/p>\n Wyj\u015bcie W drugiej kom\u00f3rce<\/p>\n Wyj\u015bcie: Przyk\u0142adowy wygl\u0105d notatnika<\/p>\n Uruchamiamy stron\u0119 http:\/\/localhost:8082<\/a> <\/p>\n <\/p>\n
\na) Virtual Box ( wirtualizator)
\nhttps:\/\/www.virtualbox.org\/wiki\/Downloads<\/a>
\nInstalujemy najwy\u017csz\u0105 wersj\u0119 , jak\u0105 aktualnie obs\u0142uguje Vagrant
\nGUI VirtualBoxa nie b\u0119dzie wog\u00f3le wykorzystywane.<\/p>\nvagrant box add http:\/\/semantica.cs.lth.se\/pyspark\/vm.json\r\n<\/pre>\n
vagrant box list\r\nlth-pysparkvm (virtualbox, 1.0.1)\r\n<\/pre>\n
vagrant status\r\nCurrent machine states:\r\nlth-pysparkvm poweroff (virtualbox)\r\nThe VM is powered off. To restart the VM, simply run `vagrant up`\r\n<\/pre>\n
vagrant up lth-pysparkvm\r\nBringing machine 'lth-pysparkvm' up with 'virtualbox' provider...\r\n==> lth-pysparkvm: Checking if box 'lth-pysparkvm' is up to date...\r\n==> lth-pysparkvm: Clearing any previously set forwarded ports...\r\n==> lth-pysparkvm: Clearing any previously set network interfaces...\r\n==> lth-pysparkvm: Preparing network interfaces based on configuration...\r\n lth-pysparkvm: Adapter 1: nat\r\n==> lth-pysparkvm: Forwarding ports...\r\n lth-pysparkvm: 8888 (guest) => 8081 (host) (adapter 1)\r\n lth-pysparkvm: 8002 (guest) => 8082 (host) (adapter 1)\r\n lth-pysparkvm: 4040 (guest) => 4040 (host) (adapter 1)\r\n lth-pysparkvm: 4041 (guest) => 4041 (host) (adapter 1)\r\n lth-pysparkvm: 4042 (guest) => 4042 (host) (adapter 1)\r\n lth-pysparkvm: 4043 (guest) => 4043 (host) (adapter 1)\r\n lth-pysparkvm: 4044 (guest) => 4044 (host) (adapter 1)\r\n lth-pysparkvm: 4045 (guest) => 4045 (host) (adapter 1)\r\n lth-pysparkvm: 22 (guest) => 2222 (host) (adapter 1)\r\n==> lth-pysparkvm: Running 'pre-boot' VM customizations...\r\n==> lth-pysparkvm: Booting VM...\r\n==> lth-pysparkvm: Waiting for machine to boot. This may take a few minutes...\r\n lth-pysparkvm: SSH address: 127.0.0.1:2222\r\n lth-pysparkvm: SSH username: vagrant\r\n lth-pysparkvm: SSH auth method: private key\r\n==> lth-pysparkvm: Machine booted and ready!\r\n==> lth-pysparkvm: Checking for guest additions in VM...\r\n lth-pysparkvm: The guest additions on this VM do not match the installed version of\r\n lth-pysparkvm: VirtualBox! In most cases this is fine, but in rare cases it can\r\n lth-pysparkvm: prevent things such as shared folders from working properly.\r\nIf you see\r\n lth-pysparkvm: shared folder errors, please make sure the guest additions within the\r\n lth-pysparkvm: virtual machine match the version of VirtualBox you have installed on\r\n lth-pysparkvm: your host and reload your VM.\r\n lth-pysparkvm:\r\n lth-pysparkvm: Guest Additions Version: 4.3.30\r\n lth-pysparkvm: VirtualBox Version: 5.1\r\n==> lth-pysparkvm: Mounting shared folders...\r\n lth-pysparkvm: \/home\/vagrant\/notebooks\/host => D:\/developing\/Vagrant\/bin\r\n\r\n\r\n<\/pre>\n
# -*- mode: ruby -*-\r\n# vi: set ft=ruby :\r\n\r\nVagrant.configure(2) do |config|\r\n config.vm.define \"lth-pysparkvm\" do |master|\r\n master.vm.box = \"lth-pysparkvm\"\r\n\r\n master.vm.network :forwarded_port, host: 8081, guest: 8888, auto_correct: true # Jupyter port\r\n master.vm.network :forwarded_port, host: 8082, guest: 8002, auto_correct: true # Updater port\r\n master.vm.network :forwarded_port, host: 4040, guest: 4040, auto_correct: true # Spark UI (Driver)\r\n master.vm.network :forwarded_port, host: 4041, guest: 4041, auto_correct: true # Spark UI (Driver)\r\n master.vm.network :forwarded_port, host: 4042, guest: 4042, auto_correct: true # Spark UI (Driver)\r\n master.vm.network :forwarded_port, host: 4043, guest: 4043, auto_correct: true # Spark UI (Driver)\r\n master.vm.network :forwarded_port, host: 4044, guest: 4044, auto_correct: true # Spark UI (Driver)\r\n master.vm.network :forwarded_port, host: 4045, guest: 4045, auto_correct: true # Spark UI (Driver)\r\n master.vm.usable_port_range = 4040..4090\r\n\r\n master.vm.provider :virtualbox do |vb|\r\n vb.name = \"lth-pysparkvm\"\r\n\r\n # Assign 2 CPUs\r\n vb.customize [\"modifyvm\", :id, \"--cpus\", \"2\"]\r\n\r\n # Assign 2 GB Memory\r\n vb.customize [\"modifyvm\", :id, \"--memory\", \"2048\"]\r\n\r\n # Disable USB\r\n vb.customize [\"modifyvm\", :id, \"--usb\", \"off\"]\r\n vb.customize [\"modifyvm\", :id, \"--usbehci\", \"off\"]\r\n end\r\n end\r\n\r\n config.vm.synced_folder \".\", \"\/home\/vagrant\/notebooks\/host\"\r\nend\r\n<\/pre>\n
\nNa stacjach z Windows 7 standardowo zainstalowany jest powershell w versji 2.0. Wykonanie polecenia Vagrant up nie ko\u0144czy si\u0119. Nale\u017cy zaktualizowa\u0107 framework np. do wersji 4.0.<\/p>\nPS C:\\Users\\User> $PSVersionTable\r\n\r\nName Value \r\n---- ----- \r\nCLRVersion 2.0.50727.8762 \r\nBuildVersion 6.1.7601.17514 \r\nPSVersion 2.0 \r\nWSManStackVersion 2.0 \r\nPSCompatibleVersions {1.0, 2.0} \r\nSerializationVersion 1.1.0.1 \r\nPSRemotingProtocolVersion 2.1 \r\n<\/pre>\nPS C:\\Users\\User> $PSVersionTable\r\nName Value \r\n---- ----- \r\nPSVersion 4.0 \r\nWSManStackVersion 3.0 \r\nSerializationVersion 1.1.0.1 \r\nCLRVersion 4.0.30319.42000 \r\nBuildVersion 6.3.9600.18728 \r\nPSCompatibleVersions {1.0, 2.0, 3.0, 4.0} \r\nPSRemotingProtocolVersion 2.2 \r\n\r\nPS C:\\Users\\User> \r\n\r\n<\/pre>\n# run net outside VM\r\n vb.customize ['modifyvm', :id, '--natdnsproxy1', 'on']\r\n vb.customize ['modifyvm', :id, '--natdnshostresolver1', 'on']\r\n vb.customize ['modifyvm', :id, '--cableconnected1', 'on']\r\n \r\n<\/pre>\n
Pierwsze kroki<\/h2>\n
1. Uruchomienie skryptu Pythona w notatniku Jupiter<\/h4>\n
#show SparkContext object\r\nsc\r\n<\/pre>\n
\nOut[1]:<\/p>\n#show type of SparkContext\r\ntype(sc)\r\n<\/pre>\n
\nOut[2]: pyspark.context.SparkContext<\/p>\n![\"Pyspark\"](\"http:\/\/195.181.217.115\/wp-content\/uploads\/2017\/11\/localhost8081_pyspark.jpg\")
<\/p>\n2. Aktualizujemy dost\u0119pne pliki z danymi<\/h4>\n
\nNaciskamy przycisk “Update all” i czekamy cierpliwie na aktualizacj\u0119, jeden z plik\u00f3w ma ponad 2GB danych.
\nPo aktualizacji wygl\u0105da to tak<\/p>\n![\"Datasets\"](\"http:\/\/195.181.217.115\/wp-content\/uploads\/2017\/11\/localhost8082_datasets-1.jpg\")
<\/p>\nDost\u0119p do UI Sparka jest\u00a0 pod adresem<\/h4>\n
![\"localhost_4040\"](\"http:\/\/195.181.217.115\/wp-content\/uploads\/2017\/11\/localhost_4040.jpg\")
<\/p>\n![\"ssh_session_pyspark\"](\"http:\/\/195.181.217.115\/wp-content\/uploads\/2017\/11\/ssh_session_pyspark.jpg\")
<\/p>\n![\"ssh_auth_pyspark\"](\"http:\/\/195.181.217.115\/wp-content\/uploads\/2017\/11\/ssh_auth_pyspark.jpg\")
<\/p>\n![\"ssh_auth_loginvagrant\"](\"http:\/\/195.181.217.115\/wp-content\/uploads\/2017\/11\/ssh_auth_loginVagrant.jpg\")
<\/p>\n