Mainframe Virtual Tape and Dataset Storage on Azure using Luminex products
Using Azure for Mainframe Virtual Tape and Dataset Storage
With Luminex products you can send/receive transmitted Mainframe data to/from Azure . In this document we will share the outcomes of the demo where we send virtual tapes, and Mainframe datasets to and from Azure.
Solution Overview:
Archiving and Recalling Virtual Tapes To Azure
- Write Tape to Luminex CGX Device
To send a virtual tape to Azure, you can simply run your normal jobs that output data to tape. The Luminex Channel Gateway (CGX) can be configured with multiple pools with different storage characteristics. For instance, one pool can be configured to send all data to Azure, while another could be configured to only use on-premise local storage. You can use SMS or Esoterics to direct jobs to the correct Luminex storage pool. In this case below, the tape X7401X is being written to device 7401 that is a part of a storage pool that sends tapes directly to Azure blob storage without keeping a copy locally. When the job is complete, the X7401X tape will only exist on Azure.
- Luminex automatically moves the tape to Azure
When the Mainframe job is completed and the tape unmounted, the Luminex devices automatically moves the data to Azure. You can view the status of each tape being sent on the Replication Monitor and in the Luminex Audit Logs. This allows you to see real time and historical status of the location of each tape. Once the movement to Azure is completed, you can view the tape’s status which will show its original size and location:
As you can see, the tape has been as been send to the azureTarget and removed from the local cache. The data was also encrypted with Luminex CGSafe encryption and compressed as it was originally written. The tape remains encrypted and compressed as it is sent to the blob storage, retaining encryption in flight and at rest.
We can also see the tape object on the Azure Blob console:
The tape, represented by the X7401X.tar object, is now on Azure and ready to be recalled as needed. In this example, the object is being stored in Hot storage tier but it could also be Cold or Archive tier based on the configuration.
Recalling a virtual tape from Azure to the Mainframe:
The process for recalling a volume back to the Mainframe can be as easy as just mounting the tape. When a Mainframe job requests a tape that is only on Azure, the Luminex CloudTAPE software will automatically retrieve it from Azure, no operator intervention required.
Here you can see that the X7401X tape as been requested for a mount. At this point the recall process will be initiated, as you can see in the CGX Monitor:
Once the recall is complete, the tape will be presented to the Mainframe and able to be read or written like any other virtual tape. If there are changes to the tape, the new version will be resent to Azure in the same way we saw above.
Sending a Mainframe Dataset to Azure
Sending virtual tapes to Azure is a great way to archive and backup data, but what if you want to share that data with other applications in Azure? In this case, it may be easier to send Mainframe dataset directly as a file to Azure. Luminex MDI can send datasets directly to Azure, and even can perform translations and conversions on the way.
Luminex MDI uses a Mainframe proc called LUMXPROC that allows you to direct Mainframe datasets through the Luminex device and send the data off the Mainframe using the native FICON I/O channel. This can be a big performance increase and MIPS savings compared to TCP/IP based transfer methods like FTP.
To use LUMXPROC, simply write the file to the Luminex device and specify any parameters needed:
In this case, the dataset coded in F1DSN will be sent to the Azure cloud. The blob name is specified in the parameters and the connection string with credentials is passed in through a secured dataset, AZS.KEYS. The resulting file on the blob storage will be named lumxdemo.csv and it will be translated from its original mainframe EBCDIC to ascii.
When the job is submitted, the MDI monitor will show the work is in progress:
After the file is converted and successfully sent to Azure, the Mainframe step will complete and continue processing. Details about the transfer will be available in the job log. At that point the file is on Azure and ready to be processed by an Azure application if needed.
Summary:
We used both Luminex’s CloudTAPE for virtual tape, and MDI for datasets, to send and receive data to Azure. This allows the Mainframe share data in a hybrid cloud environment and utilize the benefits of Azure’s blob or FS storage for its backup and archive data.
Product Overview:
Luminex’s CloudTAPE and MDI products are built on top of Luminex’s industry proven CGX virtual tape software. This software is deployed around the world with some of the largest companies and federal entities trusting it to run their critical Mainframe operations on. Using this proven technology as an interface to the Mainframe allows jobs to continue reading and writing data from its native storage interfaces as it always has. No need for zIIP engines, OSA cards, or complicated migrations to send data to Azure. Instead, just simply write the data to the Luminex and let it do the rest. Once the data is written to Luminex, it can be translated as needed (e.g. EBCDIC to ASCII or VSAM to csv), and then it is sent through to Azure. Getting data from Azure is simplified, the data is requested by the Mainframe, pulled from the Azure target to Luminex, and then presented as a dataset to the Mainframe. And because the Mainframe is just reading and writing from its native storage interface, no large CPU increases incurred from sharing data with Azure is expected. Luminex along with the below mentioned advantages highlights that 'customer’s using Luminex are reporting a 22x increased transfers speed with a 99% Mainframe CPU savings compared to sending the data with FTP. That savings has saved hybrid cloud projects that were previously missing SLAs and allowed customers to avoid massive Mainframe CPU upgrades.'
Advantages:
- Fast, Secure, and Flexible off-host data transfer
- Customers have seen 22.5 X Transfers speeds compared to Mainframe FTP.
- Built in data conversions (EBCDIC to ASCII, copybook to csv, etc..)
- Bidirectional movement for multi-platform workflows
- Easy Implementation and Operation
- No zIIPs or OSAs required.
- Can either installs alongside current VTL or replace existing VTL
- Alerts issued when needed.
- Building Block Approach
- No Limitation on Throughput.
- No Limitation on Storage Space.
- Can Add/Grow CGXs(Throughput) and/or Storage(Capacity) without impact to solution.
- Flexible Cloud Processing done off-host
- Eliminates the need for Mainframe Cycles/MSUs for processing.
- Ability to Write, Monitor and Maintain Tape Data to Multiple Sites
- No Impact to the Mainframe or Operations.
See More:
Luminex and Azure Reference Architecture - https://learn.microsoft.com/en-us/azure/architecture/example-scenario/mainframe/luminex-mainframe-file-tape-transfer
Authors:
Daniel Saunders (Sales Engineer at Luminex Software, Inc.)
Bhuvi Vatsey ( Sr. TPM at Microsoft)
Please note: The purpose of this demo is to prove how Mainframe data can be saved on Azure and recalled when needed. All the product features/ benefits have not been tested in this exercise.
