(Scala-specific) Returns a new DataFrame that drops rows containing less than
minNonNulls
non-null values in the specified columns.
(Scala-specific) Returns a new DataFrame that drops rows containing less than
minNonNulls
non-null values in the specified columns.
1.3.1
Returns a new DataFrame that drops rows containing less than minNonNulls
non-null
values in the specified columns.
Returns a new DataFrame that drops rows containing less than minNonNulls
non-null
values in the specified columns.
1.3.1
Returns a new DataFrame that drops rows containing less than minNonNulls
non-null values.
Returns a new DataFrame that drops rows containing less than minNonNulls
non-null values.
1.3.1
(Scala-specific) Returns a new DataFrame that drops rows containing null values in the specified columns.
(Scala-specific) Returns a new DataFrame that drops rows containing null values in the specified columns.
If how
is "any", then drop rows containing any null values in the specified columns.
If how
is "all", then drop rows only if every specified column is null for that row.
1.3.1
Returns a new DataFrame that drops rows containing null values in the specified columns.
Returns a new DataFrame that drops rows containing null values in the specified columns.
If how
is "any", then drop rows containing any null values in the specified columns.
If how
is "all", then drop rows only if every specified column is null for that row.
1.3.1
(Scala-specific) Returns a new that drops rows containing any null values in the specified columns.
Returns a new DataFrame that drops rows containing any null values in the specified columns.
Returns a new DataFrame that drops rows containing any null values in the specified columns.
1.3.1
Returns a new DataFrame that drops rows containing null values.
Returns a new DataFrame that drops rows containing null values.
If how
is "any", then drop rows containing any null values.
If how
is "all", then drop rows only if every column is null for that row.
1.3.1
Returns a new DataFrame that drops rows containing any null values.
Returns a new DataFrame that drops rows containing any null values.
1.3.1
(Scala-specific) Returns a new DataFrame that replaces null values.
(Scala-specific) Returns a new DataFrame that replaces null values.
The key of the map is the column name, and the value of the map is the replacement value.
The value must be of the following type: Int
, Long
, Float
, Double
, String
.
For example, the following replaces null values in column "A" with string "unknown", and null values in column "B" with numeric value 1.0.
df.na.fill(Map( "A" -> "unknown", "B" -> 1.0 ))
1.3.1
Returns a new DataFrame that replaces null values.
Returns a new DataFrame that replaces null values.
The key of the map is the column name, and the value of the map is the replacement value.
The value must be of the following type: Integer
, Long
, Float
, Double
, String
.
For example, the following replaces null values in column "A" with string "unknown", and null values in column "B" with numeric value 1.0.
import com.google.common.collect.ImmutableMap; df.na.fill(ImmutableMap.of("A", "unknown", "B", 1.0));
1.3.1
(Scala-specific) Returns a new DataFrame that replaces null values in specified string columns.
(Scala-specific) Returns a new DataFrame that replaces null values in specified string columns. If a specified column is not a string column, it is ignored.
1.3.1
Returns a new DataFrame that replaces null values in specified string columns.
Returns a new DataFrame that replaces null values in specified string columns. If a specified column is not a string column, it is ignored.
1.3.1
(Scala-specific) Returns a new DataFrame that replaces null values in specified numeric columns.
(Scala-specific) Returns a new DataFrame that replaces null values in specified numeric columns. If a specified column is not a numeric column, it is ignored.
1.3.1
Returns a new DataFrame that replaces null values in specified numeric columns.
Returns a new DataFrame that replaces null values in specified numeric columns. If a specified column is not a numeric column, it is ignored.
1.3.1
Returns a new that replaces null values in string columns with value
.
Returns a new DataFrame that replaces null values in numeric columns with value
.
Returns a new DataFrame that replaces null values in numeric columns with value
.
1.3.1
(Scala-specific) Replaces values matching keys in replacement
map.
(Scala-specific) Replaces values matching keys in replacement
map.
Key and value of replacement
map must have the same type, and can only be doubles or strings.
// Replaces all occurrences of 1.0 with 2.0 in column "height" and "weight". df.replace("height" :: "weight" :: Nil, Map(1.0 -> 2.0)); // Replaces all occurrences of "UNKNOWN" with "unnamed" in column "firstname" and "lastname". df.replace("firstname" :: "lastname" :: Nil, Map("UNKNOWN" -> "unnamed");
list of columns to apply the value replacement
value replacement map, as explained above
1.3.1
(Scala-specific) Replaces values matching keys in replacement
map.
(Scala-specific) Replaces values matching keys in replacement
map.
Key and value of replacement
map must have the same type, and can only be doubles or strings.
If col
is "*", then the replacement is applied on all string columns or numeric columns.
// Replaces all occurrences of 1.0 with 2.0 in column "height". df.replace("height", Map(1.0 -> 2.0)) // Replaces all occurrences of "UNKNOWN" with "unnamed" in column "name". df.replace("name", Map("UNKNOWN" -> "unnamed") // Replaces all occurrences of "UNKNOWN" with "unnamed" in all string columns. df.replace("*", Map("UNKNOWN" -> "unnamed")
name of the column to apply the value replacement
value replacement map, as explained above
1.3.1
Replaces values matching keys in replacement
map with the corresponding values.
Replaces values matching keys in replacement
map with the corresponding values.
Key and value of replacement
map must have the same type, and can only be doubles or strings.
import com.google.common.collect.ImmutableMap; // Replaces all occurrences of 1.0 with 2.0 in column "height" and "weight". df.replace(new String[] {"height", "weight"}, ImmutableMap.of(1.0, 2.0)); // Replaces all occurrences of "UNKNOWN" with "unnamed" in column "firstname" and "lastname". df.replace(new String[] {"firstname", "lastname"}, ImmutableMap.of("UNKNOWN", "unnamed"));
list of columns to apply the value replacement
value replacement map, as explained above
1.3.1
Replaces values matching keys in replacement
map with the corresponding values.
Replaces values matching keys in replacement
map with the corresponding values.
Key and value of replacement
map must have the same type, and can only be doubles or strings.
If col
is "*", then the replacement is applied on all string columns or numeric columns.
import com.google.common.collect.ImmutableMap; // Replaces all occurrences of 1.0 with 2.0 in column "height". df.replace("height", ImmutableMap.of(1.0, 2.0)); // Replaces all occurrences of "UNKNOWN" with "unnamed" in column "name". df.replace("name", ImmutableMap.of("UNKNOWN", "unnamed")); // Replaces all occurrences of "UNKNOWN" with "unnamed" in all string columns. df.replace("*", ImmutableMap.of("UNKNOWN", "unnamed"));
name of the column to apply the value replacement
value replacement map, as explained above
1.3.1
:: Experimental :: Functionality for working with missing data in DataFrames.
1.3.1