Setup and use CUDA and TensorFlow in Windows Subsystem for Linux 2

Table of contents

  • Install Windows preview
  • Install WSL 2 preview
  • Install Nvidia driver preview and CUDA toolkit
    • Run CUDA sample application
  • Install Docker and Nvidia container toolkit
    • Run CUDA containers
    • Troubleshoot
  • Run WSL + CUDA + Docker + Jupyter + TensorFlow
  • Encoding and decoding video with GPU in WSL?

GPU support is the most requested feature in Windows Subsystem for Linux (WSL). It is available in WSL 2.0 through Windows Insiders program. And Nvidia CUDA is supported. The following diagram shows the WDDM model supporting CUDA user mode driver running inside Linux guest:

In the Linux guest, the CUDA user mode library talks to dxgkrnl driver's /dev/dxg device using IoCtls wrapped with libdxcore API. The dxg requests then get forwarded to the Windows host system using VMBus where for those the host dxgkrnl driver makes calls to the KMD (Kernel Mode Driver) DDI handlers.

So the popular Linux AI frameworks like TensorFlow, PyTorch, etc. can work with WSL with CUDA acceleration:

The diagram shows Microsoft Windows GPU machines running on the NVIDIA hardware. For the software layers, it shows the Windows kernel, NVIDIA Windows driver, GPU virtualization, WSL2 environment (Linux kernel), NVIDIA CUDA, and other Linux AI frameworks and apps.

This article walks through the installation of Windows, WSL, CUDA in WSL, and Docker in WSL.

Install Windows preview

First, you must enable “Optional diagnostic data”, otherwise Windows cannot join Windows Insiders.

image

Then, join Windows Insiders program with Microsoft account (an account can be created if you don’t have one: https://insider.windows.com/). The channel must be Dev:

image

Then run Windows Update. It will download the pre=release installer. Windows will restart and reinstall.

Install WSL 2 preview

In Windows, make sure the following Windows features are enabled:

  • WSL:
    dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
  • Virtual machine platform:
    dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Now restart Windows, then Windows will have WSL and the wsl command line tool. Run Windows Update again to get the latest WSL 2. When this is done, in the update history, it must show 4.19.121 or later:

image

Then manually install this patch: https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi. And then run the following command as administrator to update the kernel to the latest version:

C:\WINDOWS\system32>wsl --update
Checking for updates...
Downloading updates...
Installing updates...
This change will take effect on the next full restart of WSL. To force a restart, please run 'wsl --shutdown'.
Kernel version: 5.4.72

Now go to Microsoft Store, install a Linux distribution. I installed the first one Ubuntu, it is the same as the third item Ubuntu 20.04 LTS

image

When the installation is done, use the following command to set the default version to 2 and verify:

C:\WINDOWS\system32>wsl --set-default-version 2
For information on key differences with WSL 2 please visit https://aka.ms/wsl2
The operation completed successfully.

C:\WINDOWS\system32>wsl --list --verbose
   NAME                   STATE           VERSION
* Ubuntu                 Stopped         2
   docker-desktop         Stopped         2
   docker-desktop-data    Stopped         2

WSL version can also be set for a specific distribution:

C:\WINDOWS\system32>wsl --set-version ubuntu 2
Conversion in progress, this may take a few minutes...
For information on key differences with WSL 2 please visit https://aka.ms/wsl2
The distribution is already the requested version.

Now launch the installed Ubuntu (default) from the Store, or using the command wsl:

image

For the first time launch, it asks for initializing the user name and password.

Install Nvidia driver preview and CUDA toolkit

In Windows, download the preview driver from Nvidia: https://developer.nvidia.com/cuda/wsl/download and install.

Then launch WSL, install CUDA:

apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sh -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list'
apt update
apt install -y cuda-toolkit-11-1

Run CUDA sample application

The CUDA sample source code can be downloaded from GitHub. In WSL, run:

git clone https://github.com/NVIDIA/cuda-samples.git

Then build and run a sample application

cd cuda-samples/Samples/concurrentKernels
make
./concurrentKernels

If everything is successful, it should find the GPU and show the test output:

[./concurrentKernels] - Starting...
GPU Device 0: "Turing" with compute capability 7.5

> Detected Compute SM 7.5 hardware with 20 multi-processors
Expected time for serial execution of 8 kernels = 0.080s
Expected time for concurrent execution of 8 kernels = 0.010s
Measured time for sample = 0.012s
Test passed

Install Docker and Nvidia container toolkit

First, if Docker Desktop is installed in Windows, turn off the WSL integration for WSL in the distro, because It does not work with CUDA in WSL:

image

Now go to WSL, install Docker from there:

curl https://get.docker.com | sh

Then install Nvidia container toolkit in WSL:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
curl -s -L https://nvidia.github.io/libnvidia-container/experimental/$distribution/libnvidia-container-experimental.list | sudo tee /etc/apt/sources.list.d/libnvidia-container-experimental.list
sudo apt update
sudo apt install -y nvidia-docker2

When it is done, restart docker in WSL:

sudo service docker stop
sudo service docker start

Run CUDA containers

Now it is ready to run CUDA container:

docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark

It should give output like this:

GPU Device 0: "GeForce GTX 1650 SUPER" with compute capability 7.5
> Compute 7.5 CUDA device: [GeForce GTX 1650 SUPER]
20480 bodies, total time for 10 iterations: 30.007 ms
= 139.776 billion interactions per second
= 2795.517 single-precision GFLOP/s at 20 flops per interaction

Troubleshoot

Here Windows, WSL 2, Nvidia driver are all in preview. Sometimes when you run container, docker gives error:

docker: Error response from daemon: cgroups: cannot find cgroup mount destination: unknown.
ERRO[0000] error waiting for container: context canceled

Then WSL has to be restarted. In Windows command prompt:

wsl –shutdown
wsl

If Docker gives the following error:

Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

Then install nvidia-container-toolkit:

sudo apt install -y nvidia-container-toolkit

If Docker gives error message:

Error response from daemon: cgroups: cannot find cgroup mount destination: unknown.

Then this is the fix:

sudo mkdir /sys/fs/cgroup/systemd
sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

Run WSL + CUDA + Docker + Jupyter + TensorFlow

Finally, Jupyter notebook can be used in WSL to run popular AI framework code, like TensorFlow. Nvidia’s official tutorial runs TensorFlow with Jupyter notebook in Docker image tensorflow/tensorflow:latest-gpu-py3-jupyter. However, this docker image is no longer updated and gives errors. Now there is another up-to-date image tensorflow/tensorflow:latest-gpu-jupyter. In WSL, run :

docker run -it --gpus all -p 8889:8888 tensorflow/tensorflow:latest-gpu-jupyter

Here container’s port 8888 is mapped to WSL’s port 8889, because on Windows I am already running an instance of Jupyter notebook, which already occupies port 8888.

Dock shows the following messages:

________                               _______________
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/


WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...

[I 17:15:59.736 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
jupyter_http_over_ws extension initialized. Listening on /http_over_websocket
[I 17:15:59.947 NotebookApp] Serving notebooks from local directory: /tf
[I 17:15:59.948 NotebookApp] The Jupyter Notebook is running at:
[I 17:15:59.948 NotebookApp] http://ccd3c60790ab:8888/?token=8b1969bc9a278498fd5debe119feb6d86130850166425bef
[I 17:15:59.948 NotebookApp]  or http://127.0.0.1:8888/?token=8b1969bc9a278498fd5debe119feb6d86130850166425bef
[I 17:15:59.948 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 17:15:59.953 NotebookApp]

    To access the notebook, open this file in a browser:
         file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
     Or copy and paste one of these URLs:
         http://ccd3c60790ab:8888/?token=8b1969bc9a278498fd5debe119feb6d86130850166425bef
      or http://127.0.0.1:8888/?token=8b1969bc9a278498fd5debe119feb6d86130850166425bef

Now take the last URI, replace “127.0.0.1:8888” with “localhost:8889”, go to Windows and launch browser, paste the URI: http://localhost:8889/?token=8b1969bc9a278498fd5debe119feb6d86130850166425bef, The browser should load the Jupyter notebook with built-in TensorFlow tutorial:

image

However, running the cells in the notebook, it shows some errors. If it shows an error of git not found (“FileNotFoundError: No such file or directory: 'git'”), then run these commands in a cell to install git:

!apt update
!apt install -y git

If the python code throws an error “InternalError: Blas GEMM launch failed” or “InternalError: Blas GEMV launch failed”, the fix is to run these code:

#import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

The it should work:

image

Encoding and decoding video with GPU in WSL?

Not supported yet. To test GPU encoding and decoding, the latest version of FFmpeg can be installed:

sudo add-apt-repository ppa:savoury1/graphics
sudo add-apt-repository ppa:savoury1/multimedia
sudo add-apt-repository ppa:savoury1/ffmpeg4
sudo apt-get update
sudo apt-get upgrade && sudo apt-get dist-upgrade
sudo apt-get install ffmpeg

Then install the Nvidia encoding/decoding libraries:

sudo apt install libnvidia-decode-455
sudo apt install libnvidia-encode-455

Now try to use FFmpeg to decode a video and encode it with hevc_nvenc:

ffmpeg -hwaccel auto -i input.mkv -map 0:v:0 -map 0:a -map_metadata 0 -loglevel verbose -c:v hevc_nvenc -profile:v main10 -pix_fmt yuv420p10le -preset slow -tune film -b_adapt 2 -b_ref_mode middle -bf 3 -rc vbr_hq -deblock 0:0 -rc-lookahead 25 -rc_lookahead 25 -g 250 -keyint_min 23 -refs 4 -sc_threshold 40 -temporal_aq 1 -spatial_aq 1 -nonref_p 1 -c:a aac -ar 48000 -b:a 256k -ac 6 -b:v 2048k output.nvenc.mp4 –y

It does not work and gives the following error for decoding the input video:

[h264 @ 0x55c4517dd540] decoder->cvdl->cuvidGetDecoderCaps(&caps) failed -> CUDA_ERROR_INVALID_DEVICE: invalid device ordinal
[h264 @ 0x55c4517dd540] Failed setup for format cuda: hwaccel initialisation returned error.

And it gives the following error for encoding the output video:

[hevc_nvenc @ 0x55c451065b00] OpenEncodeSessionEx failed: unsupported device (2): (no details)
[hevc_nvenc @ 0x55c451065b00] Nvenc unloaded
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

I searched around and found a video of WSLConf’s CUDA session, where Nvidia conforms GPU encoding/decoding is not yet supported in WSL, and will come in the future:

85 Comments

  • Tensorflow is amazing

  • I truly appreciate this post. I’ve been looking everywhere for this! Thank goodness I found it on Bing. <a href="https://www.totosite365.info" target="_blank" title="토토">토토</a>

  • Thanks for sharing such great information. I also write about casino& sports betting online on my personal blog. kindly check out my page and learn more about online gambling! Thanks a lot admin! <a href="https://www.badugisite.net" target="_blank" title="온라인바둑이">온라인바둑이</a>

  • i think that i saw you visited my blog so i came to “return the favor”.I’m attempting to find things to improve my site!I suppose its ok to use some of your ideas!! <a href="https://www.slotmachine777.site" target="_blank" title="릴게임">릴게임</a>

  • Many casinos even have sportsbooks, bingo and poker rooms attached. Funds can easily be transferred from one tab to another, giving real-money players in the game even more choice. Online gambling sites offer superior entertainment to playing in a land-based casino. <a href="https://www.racesite.info/" target="_blank" title="온라인경마">온라인경마</a>

  • This site is a great stepping stone for any player that is just getting started with online gambling. Armed with the right information, players can make wise decisions and start enjoying what the online casinos have to offer. <a href="https://www.powerball365.info" target="_blank" title="파워볼게임">파워볼게임</a>

  • https://businessnet.persianblog.ir/
    https://aligh593.kowsarblog.ir/
    http://aligh593.parsiblog.com/
    http://aligh593.loxblog.com/
    https://aligh593.blogsky.com/

  • <p>https://businessnet.persianblog.ir/<br />https://aligh593.kowsarblog.ir/<br />http://aligh593.parsiblog.com/<br />http://aligh593.loxblog.com/<br />https://aligh593.blogsky.com/</p>

  • Know more about this issue at https://www.dotnet.idn-kxchange.com/

  • Great delivery. Outstanding arguments. Keep up the great work. Keep posting such kind of information on your site.

  • Hello there, You have performed an excellent job. I will definitely dig it and personally suggest to my friends. I’m confident they will be benefited from this web site. <a href="https://www.bacarasite.info/" target="_blank" title="바카라">바카라</a>

  • Hi, just wanted to tell you, I loved this post. ’ve been reading your web site for a long time now, Keep on posting!

  • I am really glad to glance at this weblog posts which carries tons of helpful information, thanks for providing these data.

  • http://PhotoOnWheel.com no longer available?

  • Your article is helpful to me thanks for this helpful article. Its very interesting to read and easy to understand. Thanks for sharing.

  • Austin Curtright is a journalism sophomore and The Daily's senior sports reporter. He currently covers OU men's basketball and has previously covered OU softball and soccer.

  • Hi, ive been trying to enter the 2014 MDS but finding it difficult to get access or info from the official web site. Has anyone any ideas to help me as I don't want to miss it.

  • Four weeks left, how's the training going for everyone?

  • Thought you might like to read two articles about the MdS. The first is the participants perspective and the second is the perspective of their coach.
    https://tinyurl.com/y4fw25hp
    https://bit.ly/3a0Opei
    https://tinyurl.com/y3ufpqwy
    https://tinyurl.com/y2w3d5rz
    https://bit.ly/2MADEYg

  • You actually make it seem so easy with your presentation but I find this matter to be really something that I think I would never understand. It seems too complicated and very broad for me. I’m looking forward for your next post, I will try to get the hang of it!

  • Most of whatever you assert is astonishingly legitimate and it makes me ponder the reason why I hadn’t looked at this with this light before. This particular piece truly did turn the light on for me personally as far as this issue goes. Nevertheless there is actually one factor I am not too comfortable with so whilst I make an effort to reconcile that with the core theme of your position, allow me observe just what all the rest of the readers have to say.Well done. Visit my website too we have a lot to offer!

  • I’m really digging the template/theme of this blog. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between usability and appearance. I must say that you’ve done a fantastic job with this. Also, the blog loads very fast for me on Chrome. Excellent Blog !

  • Yeahh!!! Here I got it what I exactly needs. Such a great job!!

  • Its really a good piece of information. Glad to read it. Thank you for sharing with us.

  • Wonderful post, Thank you! This article very useful for us.

  • Very effective post, it's give me new ideas on my life.

  • awesome post. I’m a normal visitor of your web site and appreciate you taking the time to maintain the nice site. I’ll be a frequent visitor for a long time.

  • Everything is very open with a precise description of the issues. It was definitely informative. Your site is useful. Thank you for sharing!|

  • Interesting article share and this blog is impresses more people to reading that blog

  • Thanks for writing such a good article, I stumbled onto your blog and read a few post. I like your style of writing...

  • Your post is really informative and useful for everyone..

  • Great Post !! Very interesting topic will bookmark your site to check if you write more about in the future.

  • Very interesting topic will bookmark your site to check if you Post more about in the future.

  • Do you know if you still need the dev branch of Windows Insider? Dev has stability issues that broke this in the current release (21354).

  • God this documentation is awful. Pretty much the same dog shit that M$ has done for 20 years. The reason they do this is because, they don't document anything for the public, its just internal. And if you want to know how things work, well you have to hire some asshole from M$ that actually knows how the API, code, software works, because they don't document how their shit really works.

    Thanks for wasting 4 hours of my time you fucking twat........sudo make
    /usr/local/cuda/bin/nvcc -ccbin g++ -I../../Common -m64 --threads 0 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_86,code=compute_86 -o concurrentKernels.o -c concurrentKernels.cu
    nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
    /usr/local/cuda/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_86,code=compute_86 -o concurrentKernels concurrentKernels.o
    nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
    mkdir -p ../../bin/x86_64/linux/release

    So yes, fuck you and the horse you road in on.

  • thanx you admin

  • If you are going for best contents like I do, simply go to see this website all the time as it provides feature contents, thanks

  • This is very helpful, especially for those who are planning to start their new career

  • Thanks for this helpful article. Looking forward to having my portfolio. You can also read some of my great reviews about Best

  • This is very useful article. I will connect it back to your site....

  • As soon as I noticed this internet site I went on reddit to share some of the love with them. <a href="https://remarka.kz/">안전놀이터</a>

  • If you want to participate in the car electrical training course, you can refer to our school

  • If you are looking to order content production. You can refer to our site to do this with quality.

  • if u are looking for signal in digital currency u can visit our site

  • <a href="http://yun.ir/xr9boa">جراحی قلب</a> معمولا از طریق یک برش در قسمت میانی قفسه سینه و با بریدن استخوان جناغ سینه انجام میشود. بدین شکل جراح به تمام بخش ها و حفرات قلب و عروق بزرگ داخل سینه دسترسی دارد و از این طریق هرنوع عمل جراحی قلبی قابل انجام است. برای اطلاع از هزینه ها و آشنایی بیشتر با جراحی قلب به سایت دکتر بقایی مراجعه نمائید.


  • Thanks for providing great informatic and looking beautiful blog, really nice required information & the things I never imagined and I would request, write more blog and blog posts like that for us. Thank you once again.

  • I visited a lot of website but I conceive this one has something extra in it.

  • Very good written. Keep up the author held the level.

  • Nice

  • I really like this post. It is one of the best posts that I read in a long time. Thanks a lot for this really good post.

  • good information thx♥

  • آموزش نصب کولر گازی تماما بصورت عملی بوده و مخصوص ورود کارآموزان به بازار کار است.

  • در این آموزش نحوه نصب و تعمیر انواع مختلف پکیج از قبیل بوتان و ... را آموزش خواهید دید.

  • در آموزش تعمیرات موبایل به شما نحوه تعمیر انواع موبایل اندرویدی و آیوس آموزش داده خواهد شد.

  • در دوره تعمیرات برد های الکترونیکی به شما این موارد آموزش داده خواهد شد: آموزش تعمیرات بردهای لوازم خانگی ، آموزش تعمیرات برد های صنعتی ، آموزش تعمیرات برد پکیج ، آموزش تعمیرات برد کولر گازی ، آموزش تعمیرات برد اینورتر

  • طراحی و چاپ انواع ساک دستی پارچه ای و کاغذی در کوتاه ترین زمان

  • I’m not that much of a online reader to be honest but your sites really nice, keep it up!
    I’ll go ahead and bookmark your site to come back down the road.
    Cheers

  • Hi there, I found your blog by the use of Google while looking for a related subject,
    your site came up, it appears good. I've bookmarked it in my google bookmarks.

  • Hello there, just changed into aware of your blog through Google,
    and located that it's truly informative. I'm going to watch out for brussels.
    I will be grateful for those who continue this in future.
    Numerous people might be benefited out of
    your writing. Cheers!

  • Attractive component of content. I just stumbled upon your site and in accession capital to claim that I acquire actually enjoyed account
    your blog posts. Anyway I will be subscribing for your feeds or even I success you
    access consistently quickly.

  • Koch later told police he and the woman were having a dispute, and at one point, he told her "I'm going to kill you."
    <a href=" https://www.yert200.com ">카지노사이트</a>

  • Koch said he retrieved the mallet because he wanted to break down a door so he could speak to the woman.
    https://www.shine900.com

  • Police said the woman suffered four large cuts on the back of her head, a large cut on one ear, a large bump on her forehead and bruises on her left forearm.
    https://www.gain777.com

  • A Dayton police officer reported finding the body around 2:20 p.m., according to the Montgomery County Regional Dispatch Center.
    https://www.kkr789.com

  • The Montgomery County Coroner's Office also responded and left with the body shortly before 5:30 p.m. The sheriff's office said it would wait to release further information about whether the person found dead is a man or woman, how long the person had been dead and whether the death is suspicious until after the coroner's office completes its autopsy.
    https://www.aace777.com

  • The department was originally aware of 50 incidents but learned of at least 10 more after sharing information about the arrest on its Facebook page. Other people who had tires slashed Wednesday morning should call the front desk of the LPD
    https://www.qqueen700.com

  • The roommate, Jonah R. Henning, 41, is charged with misdemeanor counts of battery, disorderly conduct and bail jumping.
    https://www.rcasinosite.com

  • Henning said he was attacked when he went downstairs to start packing the washer and dryer.Henning believed his jaw was possibly broken and that he may have a concussion or head trauma.
    https://www.hgame789.com



  • I never looked at the consequences of missing a big shot... when you think about the consequences you always think of a negative result. Self-respect is the cornerstone of all virtue.
    <a href="https://www.ny-vendee.com" target="_blank">피망머니상</a>

  • Tis the most tender part of love, each other to forgive. By far the best proof is experience. Success is a lousy teacher. It seduces smart people into thinking they can't lose.
    <a href="https://www.ny-vendee.com" rel="nofollow">피망머니상</a>

  • thx for good information its really helpful.

  • Great site you have here .. It's hard to find quality writing like yours these days.
    I honestly appreciate individuals like you! Take care !!

  • Thank you for every other magnificent post. The place else could anybody get that type
    of info in such a perfect way of writing? I have a presentation subsequent week, and I'm at the
    search for such information.<a href="https://bamgosoo.com" rel="nofollow">부산오피</a><br>

  • such as Caesars Windsor and two casinos in Niagara, closed at the onset of the pandemic in
    https://www.banslot88.com/ 카지노사이트

  • while US casinos enjoyed a strong reopening – even at reduced capacities because of COVID-19 social distancing guideline https://www.cacenter77.com/ 카지노사이트

  • as the provinces did not have the same access to vaccines early in the year. That led to a prolonged wave of cases that taxed hospitals and other health resources. https://www.ggank77.com/ 카지노사이트

  • The Great Canadian venues that will reopen this Friday are: Casino Woodbine, Great Blue Heron Casino, Casino Ajax https://www.darsim77.com/ 카지노사이트

  • Elements Casino Mohawk, Elements Casino Flamboro, Elements Casino Brantford, Elements Casino Grand Rivers, Shorelines
    https://www.cadad77.com/ 카지노사이트

  • Casino Belleville, Shorelines Casino Peterborough, Shorelines Casino Thousand Islands, and Shorelines Slots at Kawartha Downs.
    https://www.cadad88.com/ 카지노사이트

  • Great Canadian Interim CEO Terrance Doyle said in a statement that the company is prepared to have thousands of workers return to their jobs.

    https://www.caxim77.com

  • I want to thank our entire team for their resilience and patience the last 17 months, and I look forward to working https://www.sans8090.com/ 카지노사이트

  • with our team and welcoming our guests back to our Ontario properties,” Doyle said https://www.bdoncasino77.com

  • As we move into the current stage of the provincial plan to reopen the economy, I’m eager to continue working https://www.cavante77.com

  • The Las Vegas Strip’s rebound from the coronavirus pandemic is coming together more rapidly tha https://www.casanta78.com

  • That’s putting the largest domestic casino hub on pace to reach pre-pandemic gaming revenue sooner than expected. https://www.dcmain77.com

Add a Comment

As it will appear on the website

Not displayed

Your website