Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- val temp = AuditorchildDF.select(AuditorchildDF.columns.map(x => col(x).as(x.replace("_", "").replace("sr:", ""))): _*)
- +-----------------+-------------------------+--------------+--------+---------+--------------------+------------------+----------------+----------------------------------+--------------------+-----------------------+-----------------------+-----------+
- |DataPartition |TimeStamp |OrganizationID|SourceID|auditorId|AuditorEnumerationId|AuditorOpinionCode|AuditorOpinionId|AuditorOpinionOnInternalControlsId|IsPlayingAuditorRole|IsPlayingCSRAuditorRole|IsPlayingTaxAdvisorRole|FFAction|!||
- +-----------------+-------------------------+--------------+--------+---------+--------------------+------------------+----------------+----------------------------------+--------------------+-----------------------+-----------------------+-----------+
- |ThirdPartyPrivate|2017-11-02T10:23:59+00:00|4298009288 |80 |16165 |null |UWE |3010547 |3020538 |true |false |true |O|!| |
- |ThirdPartyPrivate|2017-11-21T12:09:23+00:00|4295904170 |365 |3541 |3024068 |UNQ |3010546 |null |true |true |false |I|!| |
- |ThirdPartyPrivate|2017-11-21T12:17:49+00:00|4295904170 |365 |3541 |3024068 |UNQ |3010546 |null |true |true |false |O|!| |
- |ThirdPartyPrivate|2017-11-21T12:18:55+00:00|4295904170 |365 |3541 |3024068 |UNQ |3010546 |null |true |true |false |O|!| |
- |ThirdPartyPrivate|2017-05-04T09:46:05+00:00|4295907168 |447 |2815 |3023351 |UNQ |3010546 |null |true |false |false |O|!| |
- |ThirdPartyPrivate|2017-11-22T07:56:09+00:00|4295906830 |344 |3541 |3024068 |UNQ |3010546 |null |true |false |false |O|!| |
- |ThirdPartyPrivate|2017-11-22T07:56:09+00:00|4295906830 |344 |9574 |3030421 |UWE |3010547 |null |true |false |false |O|!| |
- |ThirdPartyPrivate|2018-01-11T10:00:22+00:00|5045881237 |15 |2719 |3023331 |UNQ |3010546 |null |true |false |false |I|!| |
- |ThirdPartyPrivate|2018-01-11T10:00:22+00:00|5045881237 |15 |5937 |3026578 |UWE |3010547 |null |true |false |false |I|!| |
- |ThirdPartyPrivate|2018-01-11T10:00:22+00:00|5045881237 |15 |8348 |3029157 |UNQ |3010546 |null |true |false |false |I|!| |
- |ThirdPartyPrivate|2017-11-22T07:56:09+00:00|4295906830 |344 |3541 |3024068 |UNQ |3010546 |null |true |false |false |O|!| |
- |ThirdPartyPrivate|2017-11-22T07:56:09+00:00|4295906830 |344 |9574 |3030421 |UWE |3010547 |null |true |false |false |O|!| |
- |ThirdPartyPrivate|2017-11-28T07:04:26+00:00|4295859031 |59 |3571 |3024038 |UNQ |3010546 |null |true |false |true |I|!| |
- |ThirdPartyPrivate|2017-11-28T07:04:26+00:00|4295859031 |59 |5937 |3026578 |UNQ |3010546 |null |true |false |false |I|!| |
- |ThirdPartyPrivate|2017-11-02T10:27:19+00:00|4298009288 |11 |2815 |3023351 |UNQ |3010546 |null |true |false |false |O|!| |
- |ThirdPartyPrivate|2017-11-02T10:28:25+00:00|4298009288 |13 |2815 |3023351 |UNQ |3010546 |null |true |false |false |O|!| |
- |ThirdPartyPrivate|2017-11-02T10:28:25+00:00|4298009288 |15 |2815 |3023351 |UNQ |3010546 |null |true |false |false |O|!| |
- |ThirdPartyPrivate|2018-01-11T10:01:29+00:00|5046043341 |22 |3541 |3024068 |UNQ |3010546 |null |true |false |false |O|!| |
- |ThirdPartyPrivate|2018-01-11T10:01:29+00:00|5046043341 |23 |3541 |3024068 |UNQ |3010546 |null |true |false |false |O|!| |
- |ThirdPartyPrivate|2018-01-11T10:37:51+00:00|5046043341 |218 |2749 |3023312 |UWE |3010547 |null |true |false |false |O|!| |
- +-----------------+-------------------------+--------------+--------+---------+--------------------+------------------+----------------+----------------------------------+--------------------+-----------------------+-----------------------+-----------+
- DataPartition|^|TimeStamp|^|OrganizationID|^|SourceID|^|AuditorID|^|AuditorOpinionCode|^|AuditorOpinionOnInternalControlCode|^|AuditorOpinionOnGoingConcernCode|^|IsPlayingAuditorRole|^|IsPlayingTaxAdvisorRole|^|AuditorEnumerationId|^|AuditorOpinionId|^|AuditorOpinionOnInternalControlsId|^|AuditorOpinionOnGoingConcernId|^|IsPlayingCSRAuditorRole|^|FFAction|!|
- val dfReorder = dataMain.columns.toSeq
- val diff = schema.fieldNames.diff(temp.schema.fieldNames)
- val withoutCasteddf = diff.foldLeft(temp) { (temp2df, colName) => temp2df.withColumn(colName, lit("null")) }
- And my issue and confusion
Add Comment
Please, Sign In to add comment