#11526 closed Task (fixed)
dataTranfer & clipboardData research
Reported by: | Piotr Jasiun | Owned by: | Piotr Jasiun |
---|---|---|---|
Priority: | Normal | Milestone: | CKEditor 4.5.0 Beta |
Component: | General | Version: | |
Keywords: | Cc: |
Description (last modified by )
Check what is the current status of dataTranfer & clipboardData objects in drag and drop and copy and paste.
Specs:
Attachments (6)
Change History (16)
comment:1 Changed 11 years ago by
Owner: | set to Piotr Jasiun |
---|---|
Status: | new → assigned |
comment:3 Changed 11 years ago by
1. Testing environment
I tested Drag & Drop and Copy & Paste I the following cases. You can download dnd-paste-sample.html and crayons.jpg to CKEditor's samples/ directory to test them.
HTML from outside of editable area
Source HTML:
eu odio.</p> <img src="./crayons.jpg" width="30" height="30" /> <p>Aliquam <b>mi est</b>, interdum
HTML from inside of editable area
As above.
Text from textarea
Textatea in the same document as content editable area. Textarea contains:
foo bar
2 images from system file browser
Because file API is based on files
array multiple file should works the same way as one file, but if it is not possible to drop/paste two files I also tested one file.
2 text files from system file browser
As above.
Part of image from MS Paint
Part of png/jpeg file opened with MS Paint. It is not possible to drag part of the image from Paints editing area so only copy and paste is possible. The same for Gimp.
Part of document from MS Word/Wordpad
Document containing formatted (bolded) text and image:
[b]foo[/b] [image] bar
Text from notepad
Like for Paint D&D is not possible.
Text from pdf (Adobe Reader XI)
Like for Paint D&D is not possible.
2. Results: Blink
2.1. Drag & Drop
Tested with Chrome (version 32) and Opera (version 19).
HTML from outside of editable area
Data in two types:
text/plain
: correct plain texteu odio. Aliquam mi est, interdum
text/html
: far to many html tags<p style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span class="Apple-converted-space"> </span>eu odio.</p><img src="http://ckeditor.dev/samples/local/crayons.jpg" width="30" height="30" style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;"></span><p style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Aliquam<span class="Apple-converted-space"> </span><b>mi est</b>, interdum</p>
HTML from inside of editable area
As above.
Text from textarea
Data in text/plain
.
2 images from system file browser
Chrome: Three items:
text/uri-list
with file paths- 2 files.
If I let Chrome do native D&D I get:
<a href="file:///E:/Pictures/testing%20images/butterfly.jpg">file:///E:/Pictures/testing%20images/butterfly.jpg</a>
Opera: Two file items.
If I let Opera do native D&D browsers loaded page change to
file:///C:/Users/Public/Pictures/Sample%20Pictures/Tulips.jpg
2 text files from system file browser
As above.
Part of image from MS Paint
D&D is not possible.
Part of document from MS Word/Wordpad
text/plain
: correct plain texteu odio. Aliquam mi est, interdum
text/html
: far to many html tags<p class=MsoNormal>foo<span style='mso-fareast-language:PL;mso-no-proof:yes'><!--[if gte vml 1]><v:shapetype id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f"> <v:stroke joinstyle="miter"/> <v:formulas> <v:f eqn="if lineDrawn pixelLineWidth 0"/> <v:f eqn="sum @0 1 0"/> <v:f eqn="sum 0 0 @1"/> <v:f eqn="prod @2 1 2"/> <v:f eqn="prod @3 21600 pixelWidth"/> <v:f eqn="prod @3 21600 pixelHeight"/> <v:f eqn="sum @0 0 1"/> <v:f eqn="prod @6 1 2"/> <v:f eqn="prod @7 21600 pixelWidth"/> <v:f eqn="sum @8 21600 0"/> <v:f eqn="prod @7 21600 pixelHeight"/> <v:f eqn="sum @10 21600 0"/> </v:formulas> <v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/> <o:lock v:ext="edit" aspectratio="t"/> </v:shapetype><v:shape id="Picture_x0020_1" o:spid="_x0000_i1025" type="#_x0000_t75" style='width:453pt;height:302.25pt;visibility:visible;mso-wrap-style:square'> <v:imagedata src="file:///C:\Users\pjasiun\AppData\Local\Temp\msohtmlclip1\01\clip_image001.jpg" o:title="butterfly"/> </v:shape><![endif]--><![if !vml]><img width=604 height=403 src="file:///C:\Users\pjasiun\AppData\Local\Temp\msohtmlclip1\01\clip_image002.jpg" v:shapes="Picture_x0020_1"><![endif]></span><o:p></o:p></p> <p class=MsoNormal>bar<o:p></o:p></p>
If I try native D&D (both ondrop
and ondragover
events are not canceled) I get no image and error:
Not allowed to load local resource: file:///C:/Users/pjasiun/AppData/Local/Temp/msohtmlclip1/01/clip_image002.jpg
Text from notepad
D&D is not possible.
Text from pdf (Adobe Reader XI)
D&D is not possible.
2.2. Copy & paste
For Copy and Paste Clipboard
object is available as evt.clipboardData
, not evt.dataTranfer
.
HTML from outside of editable area
The same as D&D.
HTML from inside of editable area
The same as D&D/above.
Text from textarea
The same as D&D.
2 images from system file browser
No files nor items available in clipboardData
.
2 text files from system file browser
No files nor items available in clipboardData
.
Part of image from MS Paint
One item which is file in "image/png" type (it does not matter if the source file was png, jpg or bmp).
For native drop I get nothing.
Part of document from MS Word/Wordpad
Data in three formats:
text/plain
,text/html
- is similar to dropped,text/rtf
.
If I pasted data into CKEditor I get:
<p>foo<img src="file:///C:\Users\pjasiun\AppData\Local\Temp\msohtmlclip1\01\clip_image002.jpg" style="height:403px; width:604px" /></p>
Text from notepad
Data in text/plain
.
Text from pdf (Adobe Reader XI)
Data in two formats:
text/plain
,text/rtf
.
Using native paste plain text data is inserted.
By the way: in Chrome exists additional method of DataTransferItem
: webkitGetAsEntry
next to the standard getAsFile
and getAsString
.
3. Results: Firefox
3.1. Drag & Drop
New D&D API is not available. Tested with version 26.
HTML from outside of editable area
Data in four types
text/_moz_htmlcontext
,text/_moz_htmlinfo
,text/html
: correct html<p>eu odio.</p> <img src="http://ckeditor.dev/samples/local/crayons.jpg" height="30" width="30"> <p>Aliquam <b>mi est</b>, interdum</p>
text/plain
: correct plain texteu odio. Aliquam mi est, interdum
HTML from inside of editable area
As above.
Text from textarea
Data in text/plain
.
2 images from system file browser
types
contains 3 types:
- "application/x-moz-file",
- "text/x-moz-url",
- "Files".
files
contains 2 file
elements.
2 text files from system file browser
As above.
Part of image from MS Paint
D&D is not possible.
Part of document from MS Word/Wordpad
Data in text/plain
:
Foo bar
Text from notepad
D&D is not possible.
Text from pdf (Adobe Reader XI)
D&D is not possible.
3.2. Copy & paste
for Copy and Paste Clipboard
object is available as evt.clipboardData
, not evt.dataTranfer
(as in Chrome).
HTML from outside of editable area
As D&D.
HTML from inside of editable area
As D&D.
Text from textarea
Data in text/plain
.
2 images from system file browser
types
contains 2 types:
application/x-moz-file
,Files
.
files
contains only 1 file
elements.
2 text files from system file browser
As above.
Part of image from MS Paint
No data types nor files. But you can paste image using naive paste:
<img alt="" src="data:image/png;base64,iVBORw0K…"></img>
Part of document from MS Word/Wordpad
Object as for D&D, but you can paste html (with formatting attributes, without image) using native C&P.
Also using native paste, if you select only the image in Word you can also paste it (like for part of image from MS Paint).
Text from notepad
Data in text/plain
.
Text from pdf (Adobe Reader XI)
Data in text/plain
.
4. Results: IE
4.1. Drag & Drop
For IE I tested IE 11 and IE 8.
IE does not support new API. Also it support only two data types: "Text" and "URL".
IE8 does not contains types
nor files
arrays in dataTransfer
, but methods getData()
, setData()
and clearData()
exists so you can get Text data using getData( 'Text' )
like in IE 11.
HTML from outside of editable area
You can get text data using getData("Text")
:
eu odio. Aliquam mi est, interdum
You can not get dragged HTML using DataTransfer object.
IE11: 1 type in types
: "Text".
HTML from inside of editable area
As above.
Text from textarea
Data in 1 type: "Text".
2 images from system file browser
IE11: Only "Files" data type.
2 file in
files
array.
On IE we need to cancel
dragover
event to handle files dropping. Otherwise page would be reloaded instead of calling drop event.
IE8: Drop event, but IE8 does not support file upload so getData("Text")
return null and there are no files array.
2 text files from system file browser
As above.
Part of image from MS Paint
D&D is not possible.
Part of document from MS Word/Wordpad
You can get text data using getData("Text")
:
foo bar
IE11: 1 type in types
: "Text".
If I drag single image from Word I get empty DataTransfer object (no types nor files).
Text from notepad
D&D is not possible.
Text from pdf (Adobe Reader XI)
D&D is not possible.
4.2. Copy & paste
On IE we do not have evt.clipboardData
nor evt.dataTransfer
but we can use window.clipboardData
:
evt.dataTransfer
isnull
evt.clipboardData
isundefined
window.dataTransfer
isundefined
window.clipboardData
is[object DataTransfer]
The following tests will use window.clipboardData
.
Also Microsoft Clipboard sample does not work on IE11 nor on IE8 (it works on IE 10).
HTML from outside of editable area
You can get plain text data calling clipboardData.getData("Text")
.
Using native paste you can paste html.
IE11: types
is null
.
HTML from inside of editable area
As above.
Text from textarea
As above.
2 images from system file browser
No paste event, only onBeforePaste
with no Text data nor files (on IE 11 files array exists, but contains no element). Also IE show security dialog then I use onBeforePaste
.
With native paste event nothing happen.
2 text files from system file browser
As above.
Part of image from MS Paint
IE11: One file in png format (it does not matter what was origin extension).
Using native paste you can paste image as
data:image/png;base64
.
IE8: Paste event with no text nor file.
Nothing happens using native paste.
Part of document from MS Word/Wordpad
IE11: Plain text data (clipboardData.getData("Text")
) and image file in files
array.
Using native paste you can paste proper HTML with image as
data:image/png;base64
.
IE8: Text data using getData("Text"). Plain text with native paste.
Text from notepad
You can get text data calling clipboardData.getData("Text")
.
IE11: types
is null
.
Text from pdf (Adobe Reader XI)
As above.
Native paste will insert HTML.
Changed 11 years ago by
Attachment: | drop-paste.html added |
---|
Changed 11 years ago by
Attachment: | crayons.jpg added |
---|
comment:4 Changed 11 years ago by
setData & getData
I tested if it is possible to pass data from drag event to drop event and from copy to paste event using dataTransfer
and clipboardData
. I testes if it is possible to:
- pass
text
data (usingsetData( 'text', '...' )
orsetData( 'plain/text', '...' )
), - use custom data type (using
setData( 'custom', '...' )
orsetData( 'custom/type', '...' )
), - overwrite date so browser will insert custom data on native drop or paste.
- it is not possible to handle drop event with dropbin, like paste event with pastebin. When
drop
event is called it is too late to move focus, move selection or create overall layer. Also even if we do so, we will lose drop position and need to use buggiemoveToPoint
method. Fortunately when drop method is called cursor is on drop position on IE.
Drag and Drop
text | custom | overwrite | |
---|---|---|---|
Chrome 32 | yes | yes | setData( 'plain/html' ) on dragstart |
Opera 19 | yes | yes | setData( 'plain/html' ) on dragstart |
Firefox 26 | yes | yes | setData( 'plain/html' ) on dragstart |
IE 11 | yes | no | no |
IE 10 | yes | no | no |
IE 9 | yes | no | no |
IE 8 | yes | no | no |
- for Firefox, Chrome and Opera
text
data equalstext/plain
data, so if you set both then second will overwrite first, - on IE it is not possible to replaces dragged data; it happened in widget because of bug; if you put following code in
contenteditable
area and dragimg
you will be able to droptext
data, but it works randomly and stop working is you delete any attribute.<div tabIndex=-1 contentEditable=false id="img"> <img src="./crayons.jpg" width=15 height=15> </div>
Copy and Paste
text | custom | overwrite | |
---|---|---|---|
Chrome 32 | yes | yes | setData( 'plain/html' ) on copy |
Opera 19 | yes | yes | setData( 'plain/html' ) on copy |
Firefox 26 | yes | no! | setData( 'plain/html' ) on copy |
IE 11 | yes | no | setData( 'text' ) on copy |
IE 10 | yes | no | setData( 'text' ) on copy |
IE 9 | yes | no | setData( 'text' ) on copy |
IE 8 | yes | no | setData( 'text' ) on copy |
You should call preventDefault
on copy
event to be able paste
you own data.
Changed 11 years ago by
Attachment: | setData-getData.html added |
---|
comment:5 Changed 11 years ago by
Conclusion:
- All browsers support
dataTransfer
andclipboardData
at some level. IE do it in its own way and nothing change in IE 11 (in IE 10 they added files support and this seems to be the only big change). In all browsers drop and paste works differently and different data you can read fromdataTransfer
andclipboardData
. - In all browsers we can:
- read plain text data (in both drop and paste situations),
- overwrite pasting data,
- cancel drop/paste event,
- pass some data from copy to paste and from drag to drop.
- In some (mostly IE) cases we can NOT:
- read HTML from clipboard,
- overwrite dropping data,
- use custom data types,
- drop data form image processor/pdf reader/notepad; such D&D is not supported by OS,
- use dropzone attribute.
- In "modern" browsers (IE10+, Chrome, Firefox):
- drop one or multiple files works pretty similar,
- on some browser it is possible to paste image as HTML with source data.
- Drop/Paste event should be able to contains multitype data and files. In most situation we are able to read or simulate data in both html and plain formats. It should be similar to one of the W3C APIs (new or old). It should be also backward compatible with our current paste event what might be a little complicated because it support different types but only one type at once.
- If I cancel drop event I also cancel drag event. So if you drag text form textarea and cancel drop event to handle it manually then text will be still in textarea after drop. The same for draging from Word etc.. On the other hand is not unified if data should stay in source app after Drag and Drop (ex. if you drag content form textarea in one Chromes tab to the textarea in the other tab text will not be removed from the source textarea), it depends on application.
comment:6 Changed 11 years ago by
Description: | modified (diff) |
---|
Changed 11 years ago by
Attachment: | dnd-paste-sample.html added |
---|
comment:7 Changed 11 years ago by
I created some additional summary of research.
How it is possible to get Clipboard
object?
D&D | C&P | |
---|---|---|
Blink | evt.dataTransfer (with .items) | evt.clipboardData (with .items) |
Firefox | evt.dataTransfer | evt.clipboardData |
IE 11 | evt.dataTransfer | window.clipboardData |
IE 8 | evt.dataTransfer (no .types nor .files) | window.clipboardData (no .types nor .files) |
Is it possible to read HTML?
D&D | C&P | |
---|---|---|
Blink | yes (with additional elements) | yes (with additional elements) |
Firefox | yes | yes |
IE 11 | no | no |
IE 8 | no | no |
Is it/How it is possible to get <img>
with data:image/png;base64
in src
?
D&D | C&P | |
---|---|---|
Blink | no | no |
Firefox | native drop file | native paste form MS Paint or Word (image selected) |
IE 11 | no | native paste form MS Paint or Word |
IE 8 | no | no |
Is it/How it is possible to get dropped/pasted files?
D&D | C&P | |
---|---|---|
Blink | yes, files from file system | yes, image from Paint, but files avabible only in items (files is empty)
|
Firefox | yes, files from file system | yes, files from file system (only 1 file) |
IE 11 | yes, files from file system | yes, image from Paint or from Word |
IE 8 | no | no |
Changed 11 years ago by
Attachment: | ie-movetopoint-stability.html added |
---|
IE's moveToPoint method stability tester
comment:8 Changed 11 years ago by
Milestone: | CKEditor 4.4 → CKEditor 4.5 |
---|
comment:9 Changed 10 years ago by
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
comment:10 Changed 9 years ago by
Uploading Shapes pasted from MS Word
IE11
Works.
When pasting whole word document, Chrome and FF see it as HTML and paths to images and shapes are local paths what causes security alerts. IE11 does nice conversion and all shapes can be seen as img tag with data:image string in src attribute thus images and shapes are uploaded without problems. What is more they are uploaded with their custom styles (Please try red box (with text inside) in attached shapes_test.docx).
FF
Pasting single shape works but for basic shapes only. Please have a look at attached shapes_test.docx. There are two shapes there: an arrow and red box. Arrow can be uploaded without problems while red box is seen as image with local path what prevents it from being uploaded.
Chrome
The same as in Firefox.
Chrome additionally sees "o:gfxdata" with a long string like in data:image. This is binary representation of graphics for the purposes of backward compatibility and is available only in Word 2007+. This data is zipped and base-64 encoded so perhaps something on server side could be used to unzip, decode it and extract correct shape.
Changed 9 years ago by
Attachment: | shapes_test.docx added |
---|
W3C Documentation
Both drop and paste events can contain DataTransfer object which is most important for us.
copy
,paste
etc. events asevt.clipboardData
,dragstart
,drop
etc. events asevt.dataTransfer
.DataTransfer object supports both old and new API.
New API
Basically it is a list of items (
dataTransfer.items
). Every item can be a file or a string data in certain type. Data type might be a MIME type but it is not necessary. Theoretically it is possible to have mixed files and string data. I wrote "theoretically", because it is not defined how such data should be treated:Specification does not state if data should be treated as a one piece of data in multiple formats of multiple pieces of data. It depends on data. The only information we get is: "The drag data store item list, in the order that the items were added to the list; most recently added last.".
Blink is now the only engine which supports new API.
Old API
Old API is more complex.
DataTransfer
contains:types
- an array of string e.g. ['plain/text','plain/html'] or Files?,files
- an array of files, if files array contains any element thentypes
contains 'Files' element,getData(format)
- get string data, format is a string type fromtypes
array,setData(format, data)
- set data by format.clearData(optional format)
- remove data.Old API works together with the new one so there are two ways to get data from DataTransfer object.
IE support only "Text" and "URL" types.