Hard to implement Swordfish schema in Go program language

Leon Wang
Guest

Hard to implement Swordfish schema in Go program language Sept 26, 2017 9:19:31 GMT

Quote

Post by Leon Wang on Sept 26, 2017 9:19:31 GMT

Hi everyone,

My name is Leon Wang from OpenSDS team. Currently we are trying to design OpenSDS API by using Swordfish schema, but we found it's really hard to finish the work because OpenSDS project is written in Go.

We used protobuf, which is a Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data, to define some structures and generate them into Go language. But since Go is a static-compile language so we had to define type for every variable, here is a definition of ClassOfService for example:

message ClassOfService {

string type = 1;

bytes patternProperties = 2;

bool addtionalProperties = 3;

message Properties {

Context @odata.context = 1;

Id @odata.id = 2;

Type @odata.type = 3;

message Oem {

Oem $ref = 1;

string description = 2;

string longDescription = 3;

}

message Id {

Id $ref = 1;

bool readonly = 2;

}

message Description {

oneof anyOf {

Description $ref = 1;

string type = 2;

}

bool readonly = 1;

}

message Name {

Name $ref = 1;

bool readonly = 2;

}

message Identifier {

oneof anyOf {

Identifier $ref = 1;

string type = 2;

}

bool readonly = 1;

string description = 2;

string longDescription = 3;

}

message ClassOfServiceVersion {

repeated string type = 1;

bool readonly = 2;

string description = 3;

string longDescription = 4;

}

message LinesOfService {

oneof anyOf {

LinesOfService $ref = 1;

string type = 2;

}

bool readonly = 1;

string description = 2;

string longDescription = 3;

}

}

repeated string required = 4;

string description = 5;

string longDescription = 6;

}

As you can see above, it is very complicated to define every structure using protobuf, so I'm wondering if we could simplify all Swordfish schema to make it easier for other projects to adopt it.

BR,
Leon Wang

Leon Wang
Guest

Hard to implement Swordfish schema in Go program language Sept 28, 2017 6:20:32 GMT

Quote

Post by Leon Wang on Sept 28, 2017 6:20:32 GMT

Hi all, I implemented some Swordfish schema through protobuf and created a fake Swordfish provider, if you are interested at it, please see:
github.com/opensds/opensds/tree/swordfish_implementation/contrib/swordfish

gericson
Minnow

Posts: 44

Hard to implement Swordfish schema in Go program language Sept 28, 2017 20:21:12 GMT

Quote

Post by gericson on Sept 28, 2017 20:21:12 GMT

After pretty print, your schema does't look terrible.
One change: The property names for Annotations should reflect the vocabulary schema that they belong to. You may need to add something to provide easy aliases. I Assume Context, Id, and Type are protobuf reserved words, but do you need them? I'm also not sure why you have patternProperties and additionalProperties listed at the level of ClassOfService. Note: Camel case used for Annotations. Note that properties below listed as Redfish.xxx or OData.xxx are annotations and not first class properties. Does Protobuf support annotations (i.e. metadata about the class (message). Lowercase used for OData attributes. (With apologies for syntax), For example:

message ClassOfService {
string type = 1;
// bytes patternProperties = 2;
// bool addtionalProperties = 3;
message Properties {
odata.context = 1;
odata.id = 2;
odata.type = 3;
message Oem {
Oem $ref = 1;
OData.Description = 2;
OData.LongDescription = 3;
}
message Id {
Id $ref = 1;
enum OData.Permissions readonly = 2;
bool Redfish.Required = 3
}
message Description {
oneof anyOf {
OData.Description $ref = 1;
string type = 2;
}
bool OData.Permissions readonly = 1;
}
message Name {
Name $ref = 1;
bool OData.Permissions readonly = 2;
}

message Identifier {
oneof anyOf {
Identifier $ref = 1;
string odata.type = 2;
}
bool OData.Permissions readonly = 1;
string OData.Description = 2;
string OData.LongDescription = 3;
}
message ClassOfServiceVersion {
repeated string odata.type = 1;
bool OData.Permissions readonly = 2;
string OData.Description= 3;
string OData.LongDescription= 4;
}
message LinesOfService {
oneof anyOf {
LinesOfService $ref = 1;
string odata.type = 2;
}
bool OData.Permissions readonly = 1;
string OData.Description= 2;
string OData.LongDescription= 3;
}
} repeated string required = 4;
string OData.Description = 5;
string OData.LongDescription = 6;
}

Leon Wang
Guest

Hard to implement Swordfish schema in Go program language Sept 29, 2017 6:30:52 GMT

Quote

Post by Leon Wang on Sept 29, 2017 6:30:52 GMT

Hi gericson, very appreciate it for your replies. I misunderstood the property names for annotations, so I redefined them again as below:

message Oem {}

message Identifier {
string durableName = 1;
// DurableNameFormat only contains "NAA","iQN","FC_WWN","UUID" and "EUI".
string durableNameFormat = 2;
}

message Schedule {
string name = 1;
string lifeTime = 2;
int64 maxOccurrences = 3;
string initialStartTime = 4;
string recurrenceInterval = 5;
// EnabledDaysOfWeek only supports "Monday","Tuesday","Wednesday",
// "Thursday","Friday","Saturday" and "Sunday".
repeated string enabledDaysOfWeek = 6;
// EnabledMonthsOfYear only supports "January","February","March","April",
// "May","June","July","August","September","October","November" and
// "December".
repeated string enabledMonthsOfYear = 7;
}

message Location {
string country = 1;
string territory = 2;
string state = 3;
string city = 4;
string address1 = 5;
string address2 = 6;
string address3 = 7;
string postalCode = 8;
string building = 9;
string room = 10;
string row = 11;
string rack = 12;
string shelf = 13;
string item = 14;
string gpsCoords = 15;
string otherLocationInfo = 16;
}

message IOWorkloadComponent {
// IOAccessPattern only supports "ReadWrite","SequentialRead","SequentialWrite",
// "RandomReadNew" and "RandomReadAgain".
string ioAccessPattern = 1;
int64 averageIOBytes = 2;
int64 percentOfData = 3;
int64 percentOfIOPS = 4;
string duration = 5;
Schedule schedule = 6;
}

message IOWorkload {
string name = 1;
repeated IOWorkloadComponent components = 2;
}

message DataProtectionLineOfService {
string name = 1;
// RecoveryGeographicObjective only support "Server","Rack","RackGroup",
// "Row","Datacenter" and "Region".
string recoveryGeographicObjective = 2;
string recoveryPointObjective = 3;
// RecoveryTimeObjective only supports "Immediate","Online","Nearline"
// and "Offline".
string recoveryTimeObjective = 4;
// ReplicaType only supports "Mirror","Snapshot","Clone" and
// "TokenizedClone".
string replicaType = 5;
int64 minLifetime = 6;
bool isIsolated = 7;
Schedule schedule = 8;
Location replicaAccessLocation = 9;
ClassOfService replicaClassOfService = 10;
}

message DataSecurityLineOfService {
string name = 1;
// MediaEncryptionStrength only supports "Bits_0","Bits_112","Bits_128",
// "Bits_192" and "Bits_256".
string mediaEncryptionStrength = 2;
// ChannelEncryptionStrength only supports "Bits_0","Bits_112","Bits_128",
// "Bits_192" and "Bits_256".
string channelEncryptionStrength = 3;
// HostAuthenticationType only supports "None","PKI","Ticket" and
// "Password".
string hostAuthenticationType = 4;
// UserAuthenticationType only supports "None","PKI","Ticket" and
// "Password".
string userAuthenticationType = 5;
// SecureChannelProtocol only supports "None","TLS","IPsec" and "RPCSEC_GSS".
string secureChannelProtocol = 6;
// AntivirusScanPolicies only supports "None","OnFirstRead","OnPatternUpdate",
// "OnUpdate" and "OnRename".
repeated string antivirusScanPolicies = 7;
string antivirusEngineProvider = 8;
// DataSanitizationPolicy only supports "None","Clear" and "CryptographicErase".
string dataSanitizationPolicy = 9;
}

message DataStorageLineOfService {
string name = 1;
// RecoveryTimeObjectives only supports "Immediate","Online","Nearline"
// and "Offline".
string recoveryTimeObjectives = 2;
// ProvisioningPolicy only supports "Fixed" and "Thin".
string provisioningPolicy = 3;
bool isSpaceEfficient = 4;
}

message IOConnectivityLineOfService {
string name = 1;
// AccessProtocol only supports "FC","FCP","iSCSI","SAS","SMB","NFSv3",
// "NFSv4","HTTP","HTTPS","FTP","FTPS","SecureNFS","FICON","FCOE" and
// "SMBAndNFS".
string accessProtocol = 2;
int64 maxIOPS = 3;
}

message IOPerformanceLineOfService {
string name = 1;
bool ioOperationsPerSecondIsLimited = 2;
string samplePeriod = 3;
int64 maxIOOperationsPerSecondPerTerabyte = 4;
int64 averageIOOperationLatencyMicroseconds = 5;
IOWorkload ioWorkload = 6;
}

message DataProtectionLoSCapabilitiesLinks {
Oem oem = 1;
int64 supportedReplicaOptionsOdataCount = 2;
string supportedReplicaOptionsOdataNavigationLink = 3;
repeated ClassOfService supportedReplicaOptions = 4;
}

message DataProtectionLoSCapabilities {
string odataContext = 1;
string odataId = 2;
string odataType = 3;
Oem oem = 4;
string id = 5;
string description = 6;
string name = 7;
Identifier identifier = 8;
repeated string supportedRecoveryGeographicObjectives = 9;
repeated int64 supportedRecoveryPointObjectiveSeconds = 10;
repeated string supportedRecoveryTimeObjectives = 11;
repeated string supportedReplicaTypes = 12;
repeated int64 supportedMinLifetime = 13;
bool supportsIsolated = 14;
DataProtectionLoSCapabilitiesLinks links = 15;
repeated DataProtectionLineOfService supportedDataProtectionLinesOfService = 16;
}

message DataSecurityLoSCapabilities {
string odataContext = 1;
string odataId = 2;
string odataType = 3;
Oem oem = 4;
string id = 5;
string description = 6;
string name = 7;
Identifier identifier = 8;
repeated string supportedMediaEncryptionStrengths = 9;
repeated string supportedChannelEncryptionStrengths = 10;
repeated string supportedHostAuthenticationTypes = 11;
repeated string supportedUserAuthenticationTypes = 12;
repeated string supportedSecureChannelProtocols = 13;
repeated string supportedSupportedAntivirusScanPolicies = 14;
repeated string supportedSupportedAntivirusEngineProviders = 15;
repeated string supportedDataSanitizationPolicies = 16;
repeated DataSecurityLineOfService supportedDataSecurityLinesOfService = 17;
}

message DataStorageLoSCapabilities {
string odataContext = 1;
string odataId = 2;
string odataType = 3;
Oem oem = 4;
string id = 5;
string description = 6;
string name = 7;
Identifier identifier = 8;
// AccessCapability only supports "Read","WriteMany","WriteOnce",
// "AppendOnly" and "Streaming".
repeated string supportedAccessCapabilities = 9;
repeated string supportedRecoveryTimeObjectives = 10;
repeated string supportedProvisioningPolicy = 11;
bool supportsSpaceEfficiency = 12;
repeated DataStorageLineOfService supportedDataStorageLinesOfService = 13;
}

message IOConnectivityLoSCapabilities {
string odataContext = 1;
string odataId = 2;
string odataType = 3;
Oem oem = 4;
string id = 5;
string description = 6;
string name = 7;
Identifier identifier = 8;
repeated string supportedAccessProtocols = 9;
int64 maxSupportedIOPS = 10;
repeated IOConnectivityLineOfService supportedIOConnectivityLinesOfService = 11;
}

message IOPerformanceLoSCapabilities {
string odataContext = 1;
string odataId = 2;
string odataType = 3;
Oem oem = 4;
string id = 5;
string description = 6;
string name = 7;
Identifier identifier = 8;
bool ioLimitingIsSupported = 9;
string minSamplePeriod = 10;
string maxSamplePeriod = 11;
int64 minSupportedIoOperationLatencyMicroseconds = 12;
repeated IOWorkload supportedIOWorkloads = 13;
repeated IOPerformanceLineOfService supportedIOPerformanceLinesOfService = 14;
}

message LinesOfService {
repeated DataProtectionLineOfService dataProtectionLinesOfService = 1;
repeated DataSecurityLineOfService dataSecurityLinesOfService = 2;
repeated DataStorageLineOfService dataStorageLinesOfService = 3;
repeated IOConnectivityLineOfService ioConnectivityLinesOfService = 4;
repeated IOPerformanceLineOfService ioPerformanceLinesOfService = 5;
}

// A class of service represents a choice of utility or warranty offered to
// customers by a service. NOTICE that this structure can only be called by
// storage drivers.
message ClassOfService {
string odataContext = 1;
string odataId = 2;
string odataType = 3;
Oem oem = 4 ;
string id = 5;
string description = 6;
string name = 7;
Identifier identifier = 8;
string classesOfServiceVersion = 9;
LinesOfService linesOfService = 10;
}

Here are some conclusions (gaps) from my side during implementation:

1. The first letter must be lowercase, rather than uppercase or any other characters.
2. anyOf field in Swordfish schema can not be unmarshalled into oneof field in protobuf.
3. It's very hard to implement all schema using protobuf, since it already takes almost 300 lines to define just ClassOfService schema in protobuf and the generated code lines is nearly 2000!
4. Currently many Swordfish schema has referenced Redfish schema, but some of them (actually many) are useless to OpenSDS, but we have to define them in protobuf.

That's what I got. More suggestions are welcomed

gericson
Minnow

Posts: 44

Hard to implement Swordfish schema in Go program language Sept 29, 2017 14:38:27 GMT

Quote

Post by gericson on Sept 29, 2017 14:38:27 GMT

On 1: Generally should be OK with all lowercase. Swordfish/Redfish/OData discourage name collisions based on case. Let us know if you find conflicts.

On 2: The AnyOf syntax is used in the Redfish serialization of CSDL XML to Redfish JSON. It is expressing that the property is optional (i.e. the property may have no value. In that case, a json object for a type with that property might not contain the property, or it might be returned with the special indicator of null.
For your serialization to Protobuf, consider simply ignoring the null option of the AnyOf statement.
You could alternatively consider using the ODATA serialization of CSDL XML to CSDL JSON. That serialization produces a valid, but different JSON schema representation and does not utilize the AnyOf option.
(see tools.oasis-open.org/version-control/browse/wsvn/odata/trunk/4.01 spec/examples/V4-CSDL-to-JSON.xsl) Note: This XSL is a work-in-progress.

On 3: Consider using Protobuf Package and Import statements to minimize the size of Protobuf file for a particular schema. Each CSDL XML file corresponds to a Protobuf package. You should be able to coalesce the individual CSDL Schema declarations into a single Schema for each CSDL XML file.

On 4: You don't need to include everything from Redfish or Swordfish. Each CSDL XML file contains include information that specifies the subset of schemas (packages) that need to be imported. In the Redfish JSON files that information is found in the $ref statements. In the OData JSON serializations, there are explicit $include statements.